2024-09-10T16:30:32.262 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T16:30:32.275 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T16:30:32.358 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899327 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} email: skanta@redhat.com first_in_suite: false job_id: '7899327' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgtg0j3hLU8TlYEGXHPnhncMauo7iGpAqwOrgZOohTcg+n3x50DFTjTT3iEllzvELwlOa6hf6PMOTHe7DHCHQI= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T16:30:32.359 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T16:30:32.359 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T16:30:32.359 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T16:30:32.361 INFO:teuthology.task.internal:Checking packages... 2024-09-10T16:30:32.381 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T16:30:32.381 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T16:30:32.381 INFO:teuthology.packaging:ref: None 2024-09-10T16:30:32.381 INFO:teuthology.packaging:tag: None 2024-09-10T16:30:32.381 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T16:30:32.381 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:30:32.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T16:30:32.525 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T16:30:32.526 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T16:30:32.533 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T16:30:32.533 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T16:30:32.550 INFO:teuthology.task.internal:Saving configuration 2024-09-10T16:30:32.559 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T16:30:32.566 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T16:30:32.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 16:25:33.088186', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgtg0j3hLU8TlYEGXHPnhncMauo7iGpAqwOrgZOohTcg+n3x50DFTjTT3iEllzvELwlOa6hf6PMOTHe7DHCHQI='} 2024-09-10T16:30:32.586 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T16:30:32.598 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-10T16:30:32.598 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T16:30:32.629 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9ecc347d00>, signals=[15]) 2024-09-10T16:30:32.629 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T16:30:32.635 INFO:teuthology.task.internal:Opening connections... 2024-09-10T16:30:32.635 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-09-10T16:30:32.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:30:32.713 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T16:30:32.719 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-09-10T16:30:32.794 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-09-10T16:30:32.795 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-09-10T16:30:32.853 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T16:30:32.854 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T16:30:32.855 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-09-10T16:30:32.879 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T16:30:32.887 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T16:30:32.892 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T16:30:32.892 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-09-10T16:30:32.909 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T16:30:32.915 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T16:30:32.916 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-09-10T16:30:33.072 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T16:30:33.097 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T16:30:33.113 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T16:30:33.113 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T16:30:33.114 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T16:30:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:30:33.130 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-09-10T16:30:33.188 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-503.el9.x86_64 2024-09-10T16:30:33.188 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-503.el9.x86_64 2024-09-10T16:30:33.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-09-10T16:30:35.690 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-10T16:30:37.591 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-10T16:30:43.944 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-10T16:30:45.667 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 19 kB 00:00 2024-09-10T16:30:46.554 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-10T16:30:51.985 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-10T16:30:54.017 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T16:30:54.017 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T16:30:54.113 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:30:54.124 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:30:54.124 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-09-10T16:30:54.124 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-09-10T16:30:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:30:54.126 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 79 M 2024-09-10T16:30:54.126 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 126 M 2024-09-10T16:30:54.126 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:30:54.262 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 249 kB/s | 23 kB 00:00 2024-09-10T16:30:54.679 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-10T16:30:55.104 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-09-10T16:30:55.404 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-10T16:30:55.404 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:30:55.405 INFO:teuthology.orchestra.run.smithi086.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-10T16:30:56.091 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:30:56.430 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:30:56.430 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:30:57.375 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:30:57.375 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:30:59.812 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:31:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T16:31:00.654 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T16:31:03.460 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T16:31:03.599 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T16:31:08.449 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T16:31:08.558 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:13.428 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:51.541 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:51.587 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T16:31:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T16:31:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:31:52.709 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:31:53.081 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T16:31:53.167 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T16:31:53.167 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T16:31:53.167 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T16:31:53.168 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T16:31:53.168 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T16:31:53.168 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T16:31:53.168 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T16:31:53.168 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T16:31:53.227 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T16:31:53.227 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T16:31:53.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-09-10T16:31:53.259 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-09-10T16:31:53.259 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-09-10T16:31:53.259 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-09-10T16:31:53.259 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 82.el9 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8274670 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-09-10T16:31:53.260 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:31:53.261 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T16:31:53.262 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T16:31:53.262 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-505.el9.x86_64 2024-09-10T16:31:53.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T16:31:53.922 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-09-10T16:31:55.280 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T16:31:55.314 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-09-10T16:31:55.317 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-09-10T16:31:55.333 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.SCljXITrE7 2024-09-10T16:31:55.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SCljXITrE7 2024-09-10T16:31:55.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.SCljXITrE7 2024-09-10T16:31:55.553 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.SCljXITrE7 is 6KB 2024-09-10T16:31:55.602 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.SCljXITrE7 2024-09-10T16:31:55.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T16:31:55.679 INFO:teuthology.orchestra.run.smithi086.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T16:31:55.679 INFO:teuthology.orchestra.run.smithi086.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T16:31:55.679 INFO:teuthology.orchestra.run.smithi086.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T16:31:55.680 INFO:teuthology.orchestra.run.smithi086.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T16:31:55.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T16:31:55.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-09-10T16:32:25.857 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T16:32:25.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:32:25.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:32:44.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-09-10T16:32:53.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:32:53.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:32:56.417 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-09-10T16:33:08.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:33:08.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:33:40.257 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-09-10T16:33:40.643 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:33:40.643 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T16:33:40.643 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-09-10T16:33:40.658 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-505.el9.x86_64 2024-09-10T16:33:40.659 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T16:33:40.659 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T16:33:40.659 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T16:33:41.659 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T16:33:41.660 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T16:33:41.771 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-09-10T16:33:41.791 DEBUG:teuthology.parallel:result is None 2024-09-10T16:33:41.792 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T16:33:41.799 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T16:33:41.799 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T16:33:41.814 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T16:33:41.820 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T16:33:41.835 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T16:33:41.835 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T16:33:41.893 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T16:33:41.900 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T16:33:41.900 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T16:33:41.951 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:33:41.960 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:33:41.962 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T16:33:41.969 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T16:33:41.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T16:33:42.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T16:33:42.033 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T16:33:42.033 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T16:33:42.074 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T16:33:42.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T16:33:42.225 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T16:33:42.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T16:33:42.306 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:33:42.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T16:33:42.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-09-10T16:33:42.505 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T16:33:42.637 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T16:33:42.646 INFO:teuthology.task.internal:Starting timer... 2024-09-10T16:33:42.646 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T16:33:42.657 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T16:33:42.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-09-10T16:33:42.771 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-09-10T16:33:42.775 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T16:33:42.776 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-09-10T16:33:42.828 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-09-10T16:33:42.829 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-09-10T16:33:42.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T16:33:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:33:42.898 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T16:33:42.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-09-10T16:33:43.073 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T16:33:43.262 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-10T16:33:43.266 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-10T16:33:43.400 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T16:33:43.415 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T16:33:43.416 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T16:37:25.228 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-09-10T16:37:25.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:37:25.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:37:25.314 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-09-10T16:37:25.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-10T16:37:25.388 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T16:37:25.403 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T16:37:25.403 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T16:37:25.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T16:37:25.456 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T16:37:25.470 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T16:37:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-09-10T16:37:25.505 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-09-10T16:37:25.518 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T16:37:25.532 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T16:37:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:37:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:37:25.597 INFO:teuthology.run_tasks:Running task install... 2024-09-10T16:37:25.605 DEBUG:teuthology.task.install:project ceph 2024-09-10T16:37:25.605 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T16:37:25.605 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T16:37:25.606 INFO:teuthology.task.install:Using flavor: default 2024-09-10T16:37:25.611 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T16:37:25.611 INFO:teuthology.task.install:extra packages: [] 2024-09-10T16:37:25.611 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T16:37:25.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:37:25.754 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T16:37:25.754 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T16:37:25.864 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T16:37:25.865 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:37:25.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T16:37:25.891 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T16:37:25.892 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T16:37:25.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T16:37:26.032 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T16:37:26.096 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-09-10T16:37:26.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-09-10T16:37:26.328 INFO:teuthology.orchestra.run.smithi086.stdout:45 files removed 2024-09-10T16:37:26.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-09-10T16:37:26.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 330 kB/s | 84 kB 00:00 2024-09-10T16:37:27.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 104 kB/s | 24 kB 00:00 2024-09-10T16:37:27.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-10T16:37:27.926 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-10T16:37:31.252 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 20 MB 00:02 2024-09-10T16:37:36.961 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-10T16:37:38.504 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 19 kB 00:00 2024-09-10T16:37:39.249 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 37 MB/s | 23 MB 00:00 2024-09-10T16:37:44.656 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T16:37:47.032 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T16:37:47.047 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T16:37:47.048 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:37:47.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:37:47.050 INFO:teuthology.orchestra.run.smithi086.stdout:Install 31 Packages 2024-09-10T16:37:47.051 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-09-10T16:37:47.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:37:47.051 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 62 M 2024-09-10T16:37:47.051 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:37:47.948 INFO:teuthology.orchestra.run.smithi086.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 7.5 MB/s | 5.5 MB 00:00 2024-09-10T16:37:47.982 INFO:teuthology.orchestra.run.smithi086.stdout:(2/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 761 kB/s | 25 kB 00:00 2024-09-10T16:37:48.115 INFO:teuthology.orchestra.run.smithi086.stdout:(3/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 12 MB/s | 11 MB 00:00 2024-09-10T16:37:48.149 INFO:teuthology.orchestra.run.smithi086.stdout:(4/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 4.4 MB/s | 755 kB 00:00 2024-09-10T16:37:48.174 INFO:teuthology.orchestra.run.smithi086.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 8.5 MB/s | 509 kB 00:00 2024-09-10T16:37:48.225 INFO:teuthology.orchestra.run.smithi086.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 902 kB/s | 45 kB 00:00 2024-09-10T16:37:48.433 INFO:teuthology.orchestra.run.smithi086.stdout:(7/33): ceph-common-19.3.0-4858.gbad1f558.el9.x 18 MB/s | 22 MB 00:01 2024-09-10T16:37:48.458 INFO:teuthology.orchestra.run.smithi086.stdout:(8/33): python3-ceph-common-19.3.0-4858.gbad1f5 613 kB/s | 143 kB 00:00 2024-09-10T16:37:48.484 INFO:teuthology.orchestra.run.smithi086.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 3.2 MB/s | 161 kB 00:00 2024-09-10T16:37:48.509 INFO:teuthology.orchestra.run.smithi086.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 6.2 MB/s | 321 kB 00:00 2024-09-10T16:37:48.534 INFO:teuthology.orchestra.run.smithi086.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 6.0 MB/s | 306 kB 00:00 2024-09-10T16:37:48.559 INFO:teuthology.orchestra.run.smithi086.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T16:37:48.643 INFO:teuthology.orchestra.run.smithi086.stdout:(13/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-10T16:37:48.685 INFO:teuthology.orchestra.run.smithi086.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 710 kB/s | 107 kB 00:00 2024-09-10T16:37:48.744 INFO:teuthology.orchestra.run.smithi086.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-09-10T16:37:48.786 INFO:teuthology.orchestra.run.smithi086.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 755 kB/s | 171 kB 00:00 2024-09-10T16:37:48.811 INFO:teuthology.orchestra.run.smithi086.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 956 kB/s | 160 kB 00:00 2024-09-10T16:37:48.853 INFO:teuthology.orchestra.run.smithi086.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-09-10T16:37:48.870 INFO:teuthology.orchestra.run.smithi086.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-09-10T16:37:48.895 INFO:teuthology.orchestra.run.smithi086.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-10T16:37:48.912 INFO:teuthology.orchestra.run.smithi086.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 854 kB/s | 36 kB 00:00 2024-09-10T16:37:48.929 INFO:teuthology.orchestra.run.smithi086.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-10T16:37:48.954 INFO:teuthology.orchestra.run.smithi086.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-10T16:37:49.005 INFO:teuthology.orchestra.run.smithi086.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.3 MB/s | 309 kB 00:00 2024-09-10T16:37:49.021 INFO:teuthology.orchestra.run.smithi086.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-10T16:37:49.038 INFO:teuthology.orchestra.run.smithi086.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 370 kB/s | 26 kB 00:00 2024-09-10T16:37:49.063 INFO:teuthology.orchestra.run.smithi086.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-10T16:37:49.089 INFO:teuthology.orchestra.run.smithi086.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-10T16:37:49.139 INFO:teuthology.orchestra.run.smithi086.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-10T16:37:49.189 INFO:teuthology.orchestra.run.smithi086.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-10T16:37:49.256 INFO:teuthology.orchestra.run.smithi086.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-10T16:37:49.390 INFO:teuthology.orchestra.run.smithi086.stdout:(32/33): librados2-19.3.0-4858.gbad1f558.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-10T16:37:49.448 INFO:teuthology.orchestra.run.smithi086.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-10T16:37:49.453 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:37:49.453 INFO:teuthology.orchestra.run.smithi086.stdout:Total 26 MB/s | 62 MB 00:02 2024-09-10T16:37:49.564 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:37:49.600 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:37:49.601 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:37:50.289 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:37:50.290 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:37:51.781 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:37:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T16:37:51.912 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T16:37:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T16:37:51.960 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:37:51.984 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:37:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:37:52.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:37:52.576 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T16:37:52.691 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:37:52.782 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:37:52.873 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T16:37:52.900 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:37:52.963 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:37:53.000 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T16:37:53.043 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T16:37:53.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:37:53.160 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:37:53.225 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:37:53.259 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T16:37:53.371 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T16:37:53.411 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T16:37:53.452 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T16:37:53.483 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T16:37:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T16:37:53.545 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:37:53.598 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T16:37:53.915 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T16:37:53.950 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:37:54.041 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:37:54.072 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T16:37:54.108 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T16:37:54.595 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T16:37:54.628 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:37:54.682 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:37:54.752 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T16:37:54.788 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T16:37:57.072 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:37:57.101 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:37:57.679 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:37:57.729 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:37:57.821 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:37:57.821 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T16:37:57.821 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T16:37:57.821 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:37:57.855 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:38:07.928 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-09-10T16:38:07.929 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:38:08.968 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:38:08.968 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:08.968 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T16:38:08.968 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:38:08.968 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:38:08.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:08.970 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:38:09.047 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:38:09.047 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:38:10.220 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:38:10.220 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T16:38:10.221 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T16:38:10.222 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T16:38:10.223 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T16:38:11.135 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:38:11.135 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:11.135 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-09-10T16:38:11.135 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T16:38:11.136 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T16:38:11.137 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T16:38:11.138 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T16:38:11.139 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:11.139 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:11.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-09-10T16:38:12.043 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:12.175 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:12.175 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:12.176 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:12.177 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-09-10T16:38:12.177 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:12.177 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 50 M 2024-09-10T16:38:12.177 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 215 M 2024-09-10T16:38:12.177 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:12.568 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-10T16:38:12.634 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 810 kB/s | 161 kB 00:00 2024-09-10T16:38:12.659 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 695 kB/s | 64 kB 00:00 2024-09-10T16:38:13.951 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-10T16:38:13.953 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:13.954 INFO:teuthology.orchestra.run.smithi086.stdout:Total 28 MB/s | 50 MB 00:01 2024-09-10T16:38:13.981 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:14.002 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:14.002 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:14.381 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:14.384 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:14.917 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:14.995 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T16:38:15.034 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T16:38:15.061 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:38:19.813 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:38:19.901 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:38:22.189 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:38:22.190 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T16:38:22.190 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T16:38:22.190 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T16:38:22.476 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T16:38:22.476 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:22.476 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:22.477 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:22.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-09-10T16:38:23.329 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:23.474 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:23.476 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:23.476 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:38:23.476 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T16:38:23.477 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections 2024-09-10T16:38:23.478 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T16:38:23.479 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T16:38:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T16:38:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T16:38:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T16:38:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T16:38:23.480 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T16:38:23.481 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T16:38:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:23.482 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:23.482 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:23.482 INFO:teuthology.orchestra.run.smithi086.stdout:Install 37 Packages 2024-09-10T16:38:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:23.483 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 M 2024-09-10T16:38:23.483 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 109 M 2024-09-10T16:38:23.483 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:24.293 INFO:teuthology.orchestra.run.smithi086.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 60 kB/s | 6.6 kB 00:00 2024-09-10T16:38:24.493 INFO:teuthology.orchestra.run.smithi086.stdout:(2/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 7.7 MB/s | 2.4 MB 00:00 2024-09-10T16:38:24.535 INFO:teuthology.orchestra.run.smithi086.stdout:(3/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-09-10T16:38:24.635 INFO:teuthology.orchestra.run.smithi086.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 2.8 MB/s | 291 kB 00:00 2024-09-10T16:38:24.694 INFO:teuthology.orchestra.run.smithi086.stdout:(5/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-10T16:38:24.861 INFO:teuthology.orchestra.run.smithi086.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-10T16:38:25.053 INFO:teuthology.orchestra.run.smithi086.stdout:(7/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-10T16:38:25.078 INFO:teuthology.orchestra.run.smithi086.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-10T16:38:25.111 INFO:teuthology.orchestra.run.smithi086.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.9 MB/s | 1.2 MB 00:00 2024-09-10T16:38:25.137 INFO:teuthology.orchestra.run.smithi086.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-10T16:38:25.162 INFO:teuthology.orchestra.run.smithi086.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-10T16:38:25.196 INFO:teuthology.orchestra.run.smithi086.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-10T16:38:25.221 INFO:teuthology.orchestra.run.smithi086.stdout:(13/37): python3-pycparser-2.20-6.el9.noarch.rp 806 kB/s | 135 kB 00:00 2024-09-10T16:38:25.246 INFO:teuthology.orchestra.run.smithi086.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 695 kB/s | 35 kB 00:00 2024-09-10T16:38:25.271 INFO:teuthology.orchestra.run.smithi086.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T16:38:25.296 INFO:teuthology.orchestra.run.smithi086.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-10T16:38:25.322 INFO:teuthology.orchestra.run.smithi086.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-10T16:38:25.350 INFO:teuthology.orchestra.run.smithi086.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 529 kB/s | 42 kB 00:00 2024-09-10T16:38:25.398 INFO:teuthology.orchestra.run.smithi086.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-10T16:38:25.423 INFO:teuthology.orchestra.run.smithi086.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-10T16:38:25.448 INFO:teuthology.orchestra.run.smithi086.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 446 kB/s | 43 kB 00:00 2024-09-10T16:38:25.473 INFO:teuthology.orchestra.run.smithi086.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 126 kB/s | 22 kB 00:00 2024-09-10T16:38:25.498 INFO:teuthology.orchestra.run.smithi086.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-10T16:38:25.532 INFO:teuthology.orchestra.run.smithi086.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 888 kB/s | 23 kB 00:00 2024-09-10T16:38:25.557 INFO:teuthology.orchestra.run.smithi086.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 843 kB/s | 19 kB 00:00 2024-09-10T16:38:25.582 INFO:teuthology.orchestra.run.smithi086.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-10T16:38:25.608 INFO:teuthology.orchestra.run.smithi086.stdout:(27/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-09-10T16:38:25.677 INFO:teuthology.orchestra.run.smithi086.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-10T16:38:25.677 INFO:teuthology.orchestra.run.smithi086.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-10T16:38:25.683 INFO:teuthology.orchestra.run.smithi086.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-10T16:38:25.708 INFO:teuthology.orchestra.run.smithi086.stdout:(31/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-10T16:38:25.734 INFO:teuthology.orchestra.run.smithi086.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-10T16:38:25.759 INFO:teuthology.orchestra.run.smithi086.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-10T16:38:25.784 INFO:teuthology.orchestra.run.smithi086.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T16:38:25.810 INFO:teuthology.orchestra.run.smithi086.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-10T16:38:25.826 INFO:teuthology.orchestra.run.smithi086.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-10T16:38:25.852 INFO:teuthology.orchestra.run.smithi086.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-10T16:38:25.856 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:25.856 INFO:teuthology.orchestra.run.smithi086.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-10T16:38:26.012 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:26.052 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:26.053 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:26.553 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:26.554 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:27.899 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:27.946 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T16:38:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T16:38:28.007 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T16:38:28.077 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T16:38:28.123 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T16:38:28.155 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T16:38:28.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T16:38:28.336 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T16:38:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T16:38:28.413 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T16:38:28.443 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T16:38:28.477 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T16:38:28.513 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T16:38:28.551 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T16:38:28.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T16:38:28.620 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T16:38:28.668 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T16:38:28.700 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T16:38:28.776 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T16:38:28.919 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T16:38:28.999 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T16:38:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T16:38:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T16:38:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T16:38:29.195 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T16:38:29.389 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T16:38:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T16:38:29.519 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T16:38:29.602 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T16:38:29.649 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T16:38:29.779 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T16:38:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T16:38:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:38:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:31.461 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:38:31.489 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:38:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T16:38:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:38:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:38:31.490 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:31.937 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:38:31.961 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:32.188 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:38:32.211 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:38:32.212 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:32.212 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T16:38:32.212 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:38:32.212 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:38:32.212 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:32.276 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:38:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:38:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T16:38:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T16:38:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T16:38:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T16:38:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T16:38:34.881 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:34.882 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T16:38:34.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T16:38:34.884 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:34.885 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:35.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-09-10T16:38:35.793 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:35.883 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:38:35.925 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:35.926 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:38:35.926 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:35.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-09-10T16:38:36.550 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:36.680 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:36.681 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:38:36.682 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:36.682 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 784 k 2024-09-10T16:38:36.682 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 790 k 2024-09-10T16:38:36.682 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:36.961 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-10T16:38:36.961 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:36.962 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-10T16:38:36.962 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:36.967 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:36.967 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:36.973 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:36.973 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:37.047 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:37.548 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:38:37.631 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:38:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:38:38.236 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:38:38.237 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:38.237 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:38.237 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:38:38.237 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:38.237 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:38.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T16:38:39.023 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:39.157 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:39.157 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:39.157 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-09-10T16:38:39.157 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:39.158 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2024-09-10T16:38:39.159 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 432 k 2024-09-10T16:38:39.159 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:39.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 799 kB/s | 145 kB 00:00 2024-09-10T16:38:39.342 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:39.343 INFO:teuthology.orchestra.run.smithi086.stdout:Total 792 kB/s | 145 kB 00:00 2024-09-10T16:38:39.343 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:39.350 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:39.350 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:39.407 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:39.408 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:39.565 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:39.642 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:38:39.667 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:38:39.667 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:38:39.667 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T16:38:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:40.509 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:38:40.510 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:40.510 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:40.510 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:38:40.510 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:40.510 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:40.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-09-10T16:38:41.315 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:41.409 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:38:41.450 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:41.451 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:38:41.451 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:41.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T16:38:42.090 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:42.224 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:42.224 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T16:38:42.225 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-09-10T16:38:42.226 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:42.227 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 19 M 2024-09-10T16:38:42.227 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 159 M 2024-09-10T16:38:42.227 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:42.611 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 224 kB/s | 34 kB 00:00 2024-09-10T16:38:42.628 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 97 kB/s | 16 kB 00:00 2024-09-10T16:38:43.095 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f55 22 MB/s | 14 MB 00:00 2024-09-10T16:38:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-09-10T16:38:43.470 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-10T16:38:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-10T16:38:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-protobuf-3.14.0-13.el9.noarch.r 304 kB/s | 269 kB 00:00 2024-09-10T16:38:43.562 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-10T16:38:43.588 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-10T16:38:43.621 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-10T16:38:43.672 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-10T16:38:43.897 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 803 kB/s | 1.0 MB 00:01 2024-09-10T16:38:43.903 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:43.903 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-10T16:38:43.956 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:43.977 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:43.977 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:44.258 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:44.258 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:44.868 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:44.956 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T16:38:44.991 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T16:38:45.037 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T16:38:45.068 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T16:38:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T16:38:45.361 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T16:38:45.462 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T16:38:45.541 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T16:38:45.578 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:38:45.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T16:38:48.793 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T16:38:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:38:50.717 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:38:50.717 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T16:38:50.717 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T16:38:50.717 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T16:38:50.717 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T16:38:50.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T16:38:51.027 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T16:38:51.027 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:51.027 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:38:51.027 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:51.028 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:38:51.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T16:38:51.860 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:07 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:38:51.994 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:38:51.995 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T16:38:51.996 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-09-10T16:38:51.997 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-09-10T16:38:51.998 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:38:51.998 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 38 M 2024-09-10T16:38:51.998 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 212 M 2024-09-10T16:38:51.998 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:38:52.378 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-10T16:38:52.445 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-10T16:38:52.512 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-10T16:38:52.554 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-10T16:38:52.613 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-10T16:38:52.688 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-10T16:38:52.788 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): ceph-mgr-diskprediction-local-19.3.0-48 13 MB/s | 7.4 MB 00:00 2024-09-10T16:38:52.813 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-10T16:38:52.855 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-10T16:38:52.947 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 15 MB/s | 4.9 MB 00:00 2024-09-10T16:38:53.181 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-09-10T16:38:53.273 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-10T16:38:53.274 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:38:53.274 INFO:teuthology.orchestra.run.smithi086.stdout:Total 30 MB/s | 38 MB 00:01 2024-09-10T16:38:53.632 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:38:53.657 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:38:53.658 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:38:54.076 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:38:54.076 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:38:54.830 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:38:54.910 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T16:38:54.954 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:38:55.052 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:38:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T16:38:55.275 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T16:38:55.304 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T16:38:55.381 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T16:38:56.038 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T16:38:56.152 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T16:38:57.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:38:58.487 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T16:38:58.512 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:39:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:39:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T16:39:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:39:01.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:39:01.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T16:39:01.570 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:01.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-09-10T16:39:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:18 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T16:39:02.630 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-09-10T16:39:02.632 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.2 M 2024-09-10T16:39:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-09-10T16:39:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 215 kB/s | 19 kB 00:00 2024-09-10T16:39:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 224 kB/s | 26 kB 00:00 2024-09-10T16:39:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 285 kB/s | 49 kB 00:00 2024-09-10T16:39:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-10T16:39:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-10T16:39:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-10T16:39:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 282 kB/s | 14 kB 00:00 2024-09-10T16:39:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.6 MB/s | 220 kB 00:00 2024-09-10T16:39:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-10T16:39:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 12 MB/s | 1.0 MB 00:00 2024-09-10T16:39:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-cachetools-4.2.4-1.el9.noarch. 153 kB/s | 32 kB 00:00 2024-09-10T16:39:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 552 kB/s | 59 kB 00:00 2024-09-10T16:39:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-10T16:39:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-10T16:39:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:04.115 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:04.293 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T16:39:04.348 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T16:39:04.411 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T16:39:04.449 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T16:39:04.489 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T16:39:04.590 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T16:39:04.693 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T16:39:04.741 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T16:39:05.453 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T16:39:05.505 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T16:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T16:39:05.587 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T16:39:05.609 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:39:06.397 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:39:06.397 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T16:39:06.397 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T16:39:06.397 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T16:39:06.397 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T16:39:06.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T16:39:06.762 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T16:39:06.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T16:39:06.764 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T16:39:06.764 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:06.764 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:07.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T16:39:07.575 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:23 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:07.708 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T16:39:07.709 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-09-10T16:39:07.710 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:07.711 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 7.0 M 2024-09-10T16:39:07.711 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-09-10T16:39:07.711 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:08.197 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-10T16:39:08.222 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 684 kB/s | 154 kB 00:00 2024-09-10T16:39:08.273 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T16:39:08.298 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-10T16:39:08.331 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-10T16:39:09.440 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.1 MB/s | 6.0 MB 00:01 2024-09-10T16:39:09.442 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:09.443 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-10T16:39:09.513 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:09.524 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:09.525 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:09.624 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:09.625 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:09.961 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T16:39:10.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T16:39:10.714 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T16:39:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T16:39:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T16:39:10.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:39:11.629 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:39:11.630 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T16:39:11.630 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T16:39:11.630 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T16:39:11.630 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T16:39:11.630 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T16:39:11.886 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T16:39:11.886 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T16:39:11.887 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:11.888 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:12.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-09-10T16:39:12.696 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:28 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:12.828 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-09-10T16:39:12.830 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:12.830 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 981 k 2024-09-10T16:39:12.830 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.9 M 2024-09-10T16:39:12.830 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:13.158 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.2 MB/s | 901 kB 00:00 2024-09-10T16:39:13.600 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 111 kB/s | 80 kB 00:00 2024-09-10T16:39:13.601 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:13.601 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 981 kB 00:00 2024-09-10T16:39:13.613 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:13.637 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:13.637 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:13.719 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:13.720 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:13.907 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:14.052 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T16:39:14.115 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:39:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:39:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:15.018 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:15.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-09-10T16:39:15.890 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:16.026 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:16.027 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:16.028 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:16.028 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 285 k 2024-09-10T16:39:16.028 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.3 M 2024-09-10T16:39:16.028 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:16.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-10T16:39:16.254 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:16.254 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-10T16:39:16.254 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:16.262 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:16.263 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:16.293 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:16.294 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:16.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:39:16.589 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:39:16.589 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:39:16.589 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T16:39:16.589 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:17.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:39:17.596 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:17.596 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:17.597 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:39:17.597 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:17.597 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:17.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-09-10T16:39:18.469 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:18.605 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:18.606 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:18.606 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-09-10T16:39:18.606 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:18.606 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:18.606 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 126 k 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-09-10T16:39:18.607 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:18.807 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 632 kB/s | 126 kB 00:00 2024-09-10T16:39:18.808 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:18.808 INFO:teuthology.orchestra.run.smithi086.stdout:Total 628 kB/s | 126 kB 00:00 2024-09-10T16:39:18.808 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:18.814 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:18.814 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:18.877 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:18.878 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:19.031 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:19.140 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:19.531 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:19.782 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:20.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-09-10T16:39:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:20.672 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:20.718 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:20.719 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:20.719 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:20.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-09-10T16:39:21.359 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:21.491 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T16:39:21.492 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 32 k 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 142 k 2024-09-10T16:39:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:21.644 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 210 kB/s | 32 kB 00:00 2024-09-10T16:39:21.644 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:21.645 INFO:teuthology.orchestra.run.smithi086.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-10T16:39:21.645 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:21.648 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:21.648 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:21.665 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:21.666 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:21.743 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:21.866 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:22.239 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:22.460 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:22.461 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:22.461 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:22.461 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:39:22.461 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:22.461 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:22.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-09-10T16:39:23.197 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:23.287 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:23.330 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:23.331 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:23.331 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:23.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-09-10T16:39:23.971 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:24.060 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:24.103 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:24.104 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:24.104 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:24.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-09-10T16:39:24.732 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:24.821 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:24.864 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:24.864 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:24.865 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:24.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-09-10T16:39:25.504 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:25.593 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:25.637 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:25.638 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:25.638 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:25.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-09-10T16:39:26.286 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:26.383 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:26.428 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:26.429 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:26.429 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:26.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-09-10T16:39:27.103 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:27.197 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:39:27.242 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:27.243 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-09-10T16:39:27.243 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:27.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-09-10T16:39:27.880 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:28.014 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:28.015 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:28.016 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2024-09-10T16:39:28.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2024-09-10T16:39:28.016 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:28.205 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 449 kB/s | 85 kB 00:00 2024-09-10T16:39:28.206 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:28.206 INFO:teuthology.orchestra.run.smithi086.stdout:Total 445 kB/s | 85 kB 00:00 2024-09-10T16:39:28.206 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:28.214 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:28.214 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:28.278 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:28.278 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:28.431 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:28.896 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:29.132 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:29.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-09-10T16:39:29.971 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:30.110 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:30.111 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.2 M 2024-09-10T16:39:30.112 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 14 M 2024-09-10T16:39:30.112 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:30.467 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.0 MB/s | 3.2 MB 00:00 2024-09-10T16:39:30.468 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:30.468 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.9 MB/s | 3.2 MB 00:00 2024-09-10T16:39:30.468 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:30.477 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:30.477 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:30.563 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:30.563 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:30.999 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:31.059 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:39:31.085 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:32.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:32.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:32.142 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:32.142 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:39:32.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:32.143 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:32.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-09-10T16:39:32.969 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 04:37:44 PM UTC. 2024-09-10T16:39:33.102 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:33.103 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 170 k 2024-09-10T16:39:33.104 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 490 k 2024-09-10T16:39:33.104 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-09-10T16:39:33.294 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 896 kB/s | 170 kB 00:00 2024-09-10T16:39:33.295 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:39:33.295 INFO:teuthology.orchestra.run.smithi086.stdout:Total 889 kB/s | 170 kB 00:00 2024-09-10T16:39:33.295 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-09-10T16:39:33.302 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-09-10T16:39:33.302 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-09-10T16:39:33.367 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-09-10T16:39:33.367 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-09-10T16:39:33.523 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-09-10T16:39:33.632 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:34.305 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:39:34.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:34.305 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-09-10T16:39:34.305 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:39:34.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-10T16:39:34.306 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-09-10T16:39:34.531 DEBUG:teuthology.parallel:result is None 2024-09-10T16:39:34.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:39:34.673 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T16:39:34.695 INFO:teuthology.orchestra.run.smithi086.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T16:39:34.696 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T16:39:34.696 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T16:39:34.698 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T16:39:34.698 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:39:34.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T16:39:34.760 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T16:39:34.761 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:39:34.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T16:39:34.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T16:39:34.890 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T16:39:34.890 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:39:34.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T16:39:34.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T16:39:35.019 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T16:39:35.019 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:39:35.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T16:39:35.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T16:39:35.147 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T16:39:35.163 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:39:35.163 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T16:39:35.163 INFO:tasks.workunit:timeout=3h 2024-09-10T16:39:35.163 INFO:tasks.workunit:cleanup=True 2024-09-10T16:39:35.164 DEBUG:teuthology.orchestra.run.smithi086:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T16:39:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:39:35.203 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T16:39:35.203 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T16:39:35.258 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T16:39:35.258 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T16:39:35.313 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:39:35.370 INFO:tasks.workunit.client.0.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T16:40:25.037 INFO:tasks.workunit.client.0.smithi086.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr: git switch -c 2024-09-10T16:40:25.038 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr:Or undo this operation with: 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr: git switch - 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr: 2024-09-10T16:40:25.039 INFO:tasks.workunit.client.0.smithi086.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T16:40:25.041 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T16:40:25.096 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-10T16:40:25.097 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T16:40:25.153 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-10T16:40:25.153 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-10T16:40:25.154 DEBUG:teuthology.orchestra.run.smithi086:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-10T16:40:25.228 INFO:tasks.workunit.client.0.smithi086.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-10T16:40:25.228 INFO:tasks.workunit.client.0.smithi086.stderr:++ TIMEOUT=300 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ MAX_TIMEOUT=15 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ PG_NUM=4 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ TMPDIR=/tmp 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:+++ uname 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' Linux = FreeBSD ']' 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ SED=sed 2024-09-10T16:40:25.229 INFO:tasks.workunit.client.0.smithi086.stderr:++ AWK=awk 2024-09-10T16:40:25.230 INFO:tasks.workunit.client.0.smithi086.stderr:+++ stty -a 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:+++ head -1 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ termwidth= 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ DIFFCOLOPTS='-y ' 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ KERNCORE=kernel.core_pattern 2024-09-10T16:40:25.231 INFO:tasks.workunit.client.0.smithi086.stderr:++ EXTRA_OPTS= 2024-09-10T16:40:25.233 INFO:tasks.workunit.client.0.smithi086.stderr:++ test '' = TESTS 2024-09-10T16:40:25.234 INFO:tasks.workunit.client.0.smithi086.stderr:++ uname -m 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ arch=x86_64 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ case $arch in 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ main test-erasure-code-plugins 2024-09-10T16:40:25.237 INFO:tasks.workunit.client.0.smithi086.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:+ shift 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:+ shopt -s -o xtrace 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T16:40:25.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code-plugins 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-10T16:40:25.239 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-10T16:40:25.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none ' 2024-09-10T16:40:25.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-10T16:40:25.240 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-10T16:40:25.240 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T16:40:25.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-10T16:40:25.242 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_preload_no_warning 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_preload_no_warning_default 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_preload_warning' 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:40:25.243 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:40:25.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:40:25.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:40:25.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:40:25.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:40:25.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:40:25.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:40:25.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:40:25.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:40:25.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:40:25.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:40:25.246 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:40:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:40:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:40:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:40:25.247 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:40:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:40:25.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:40:25.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:40:25.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:40:25.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:25.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:25.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:40:25.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:40:25.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:40:25.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:40:25.252 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:40:25.253 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:25.253 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:25.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:40:25.254 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:40:25.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T16:40:25.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T16:40:25.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:40:25.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:25.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:40:25.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T16:40:25.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:40:25.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:40:25.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:40:25.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:40:25.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:40:25.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:40:25.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:40:25.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:40:25.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:40:25.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:40:25.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:40:25.917 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:40:25.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:40:25.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:25.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:40:25.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:40:25.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:40:25.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:40:25.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:26.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:40:26.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:40:26.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:40:26.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:40:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:40:26.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:40:26.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a9c1d320-ce2c-48d8-9482-562203600aaf 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a9c1d320-ce2c-48d8-9482-562203600aaf' 2024-09-10T16:40:26.874 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 a9c1d320-ce2c-48d8-9482-562203600aaf 2024-09-10T16:40:26.875 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:40:26.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6duBm02HaNBAAw7QvRFhapEwHbiYqhwfNuQ== 2024-09-10T16:40:26.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6duBm02HaNBAAw7QvRFhapEwHbiYqhwfNuQ=="}' 2024-09-10T16:40:26.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a9c1d320-ce2c-48d8-9482-562203600aaf -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:40:27.168 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:40:27.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:40:27.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6duBm02HaNBAAw7QvRFhapEwHbiYqhwfNuQ== --osd-uuid a9c1d320-ce2c-48d8-9482-562203600aaf 2024-09-10T16:40:27.231 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:27.229+0000 7f5b6d022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:27.236 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:27.235+0000 7f5b6d022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:27.237 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:27.236+0000 7f5b6d022640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:27.238 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:27.236+0000 7f5b6d022640 -1 bdev(0x557c067fdc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:40:27.238 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:27.236+0000 7f5b6d022640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:40:33.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:40:33.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:40:33.164 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:40:33.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:40:33.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:40:33.688 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:40:33.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:40:33.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:40:33.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:40:33.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:40:33.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:40:33.713 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:33.711+0000 7f713c8da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:33.729 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:33.727+0000 7f713c8da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:33.730 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:33.729+0000 7f713c8da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:34.134 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:40:34.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:40:34.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:34.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:34.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:35.079 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:35.078+0000 7f713c8da640 -1 Falling back to public interface 2024-09-10T16:40:35.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:35.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:35.553 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:40:35.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:40:35.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:35.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:35.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:36.914 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:40:36.913+0000 7f713c8da640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:40:36.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:36.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:36.979 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:40:36.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:40:36.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:36.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:37.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:38.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:38.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:38.510 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:40:38.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:40:38.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:38.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:38.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:39.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:39.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:39.930 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:40:39.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:40:39.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:39.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:41.322 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:40:41.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:40:41.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:41.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:41.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:42.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:42.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:42.777 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:40:42.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:40:42.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:42.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:43.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:44.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:44.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:44.174 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:40:44.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:40:44.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:44.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:44.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:45.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:45.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:45.628 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:40:45.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:40:45.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:45.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:46.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:47.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:47.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:47.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:40:47.044 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:40:47.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:47.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:47.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:48.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:48.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:48.449 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:40:48.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:40:48.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:48.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:48.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:49.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:49.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:49.988 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:40:49.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:40:49.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:49.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:50.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:51.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:51.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:51.538 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:40:51.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:40:51.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:51.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:52.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:53.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:53.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:53.065 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:40:53.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:40:53.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:53.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:53.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:54.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:54.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:54.769 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:40:54.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:40:54.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:54.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:55.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:56.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:56.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:56.684 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:40:56.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:40:56.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:56.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:57.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:40:58.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:40:58.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:40:58.390 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:40:58.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:40:58.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:40:58.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:40:59.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:00.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:00.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:00.824 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:41:00.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:41:00.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:00.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:41:01.309 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1311997108,v1:127.0.0.1:6803/1311997108] [v2:127.0.0.1:6804/1311997108,v1:127.0.0.1:6805/1311997108] exists,up a9c1d320-ce2c-48d8-9482-562203600aaf 2024-09-10T16:41:01.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:41:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:41:01.315 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:41:01.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=82d2c446-1686-4fb3-bbdc-0f78244716a9 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 82d2c446-1686-4fb3-bbdc-0f78244716a9 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 82d2c446-1686-4fb3-bbdc-0f78244716a9' 2024-09-10T16:41:01.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:41:01.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdduBmwM+SExAAc9gZIJyC2VriErony8i/ig== 2024-09-10T16:41:01.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdduBmwM+SExAAc9gZIJyC2VriErony8i/ig=="}' 2024-09-10T16:41:01.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 82d2c446-1686-4fb3-bbdc-0f78244716a9 -i td/test-erasure-code-plugins/1/new.json 2024-09-10T16:41:01.811 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:41:01.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-10T16:41:01.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdduBmwM+SExAAc9gZIJyC2VriErony8i/ig== --osd-uuid 82d2c446-1686-4fb3-bbdc-0f78244716a9 2024-09-10T16:41:01.849 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:01.848+0000 7ffa41028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:01.851 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:01.850+0000 7ffa41028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:01.852 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:01.851+0000 7ffa41028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:01.853 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:01.851+0000 7ffa41028640 -1 bdev(0x555dac0c7c00 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2024-09-10T16:41:01.853 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:01.851+0000 7ffa41028640 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-10T16:41:07.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-10T16:41:07.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:41:07.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T16:41:07.293 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T16:41:07.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:41:07.817 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T16:41:07.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T16:41:07.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:41:07.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:41:07.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:41:07.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:41:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:07.841+0000 7f86f069c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:07.856 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:07.855+0000 7f86f069c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:07.858 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:07.856+0000 7f86f069c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T16:41:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:41:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T16:41:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:08.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:08.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:08.967 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:08.966+0000 7f86f069c640 -1 Falling back to public interface 2024-09-10T16:41:09.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:09.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:09.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:41:09.665 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:41:09.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:09.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:10.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:10.492 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:10.491+0000 7f86f069c640 -1 osd.1 0 log_to_monitors true 2024-09-10T16:41:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:11.053 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:41:11.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:41:11.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:11.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:11.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:12.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:12.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:12.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:41:12.648 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:41:12.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:12.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:13.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:14.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:14.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:14.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:41:14.206 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:41:14.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:14.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:14.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:15.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:15.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:15.641 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:41:15.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:41:15.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:15.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:16.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:17.036 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:41:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:41:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:17.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:18.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:18.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:18.661 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:41:18.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:41:18.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:18.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:19.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:20.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:20.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:20.153 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:41:20.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:41:20.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:20.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:20.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:21.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:21.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:21.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:41:21.645 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:41:21.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:21.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:22.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:23.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:23.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:23.056 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:41:23.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:41:23.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:23.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:23.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:24.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:24.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:24.820 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:41:24.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:41:24.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:24.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:26.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:27.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:27.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:27.017 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:41:27.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:41:27.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:27.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:27.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:28.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:28.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:28.870 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:41:28.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:41:28.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:28.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:29.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:30.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:30.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:30.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:41:30.631 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:41:30.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:30.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:31.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:32.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:32.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:32.559 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:41:32.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:41:32.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:32.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:33.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:34.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:34.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:34.471 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:41:34.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:41:34.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:34.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:41:34.862 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1371750528,v1:127.0.0.1:6811/1371750528] [v2:127.0.0.1:6812/1371750528,v1:127.0.0.1:6813/1371750528] exists,up 82d2c446-1686-4fb3-bbdc-0f78244716a9 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T16:41:34.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:41:34.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:41:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da6680c0-2b29-4533-91e0-bc49376b8d49 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 da6680c0-2b29-4533-91e0-bc49376b8d49' 2024-09-10T16:41:34.869 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 da6680c0-2b29-4533-91e0-bc49376b8d49 2024-09-10T16:41:34.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:41:34.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+duBmAzCMNBAA5yVGE0CRY0uDJGc7+5kf8w== 2024-09-10T16:41:34.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+duBmAzCMNBAA5yVGE0CRY0uDJGc7+5kf8w=="}' 2024-09-10T16:41:34.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da6680c0-2b29-4533-91e0-bc49376b8d49 -i td/test-erasure-code-plugins/2/new.json 2024-09-10T16:41:35.337 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:41:35.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-10T16:41:35.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+duBmAzCMNBAA5yVGE0CRY0uDJGc7+5kf8w== --osd-uuid da6680c0-2b29-4533-91e0-bc49376b8d49 2024-09-10T16:41:35.375 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:35.374+0000 7f84e421e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:35.390 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:35.389+0000 7f84e421e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:35.391 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:35.390+0000 7f84e421e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:35.392 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:35.390+0000 7f84e421e640 -1 bdev(0x55556d90dc00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2024-09-10T16:41:35.392 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:35.390+0000 7f84e421e640 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-10T16:41:40.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-10T16:41:40.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:41:40.786 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T16:41:40.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T16:41:40.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:41:41.315 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T16:41:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T16:41:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:41:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:41:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:41:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:41:41.341 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:41.339+0000 7f1cef71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:41.361 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:41.359+0000 7f1cef71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:41.362 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:41.360+0000 7f1cef71b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:41:41.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T16:41:41.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:41:41.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:41:41.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:41.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:42.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:42.210 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:42.209+0000 7f1cef71b640 -1 Falling back to public interface 2024-09-10T16:41:43.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:43.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:43.185 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:41:43.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:41:43.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:43.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:43.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:44.130 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:41:44.129+0000 7f1cef71b640 -1 osd.2 0 log_to_monitors true 2024-09-10T16:41:44.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:44.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:44.605 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:41:44.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:41:44.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:44.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:45.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:46.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:46.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:46.055 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:41:46.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:41:46.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:46.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:46.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:47.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:47.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:47.488 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:41:47.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:41:47.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:47.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:47.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:48.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:48.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:48.966 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:41:48.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:41:48.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:48.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:49.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:50.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:50.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:50.525 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:41:50.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:41:50.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:50.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:51.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:52.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:52.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:52.026 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:41:52.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:41:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:52.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:53.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:53.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:53.470 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:41:53.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:41:53.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:53.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:53.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:54.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:54.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:54.952 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:41:54.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:41:54.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:54.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:55.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:56.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:56.370 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:41:56.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:41:56.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:56.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:56.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:57.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:41:58.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:41:59.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:41:59.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:41:59.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:41:59.372 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:41:59.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:41:59.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:00.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:01.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:01.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:01.127 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:42:01.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:42:01.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:01.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:01.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:02.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:42:02.942 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:42:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:03.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:04.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:04.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:04.800 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:42:04.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:42:04.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:04.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:05.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:06.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:06.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:06.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:42:06.635 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:42:06.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:06.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:07.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:08.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:08.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:08.373 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:42:08.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:42:08.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:08.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:42:08.787 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/544598742,v1:127.0.0.1:6819/544598742] [v2:127.0.0.1:6820/544598742,v1:127.0.0.1:6821/544598742] exists,up da6680c0-2b29-4533-91e0-bc49376b8d49 2024-09-10T16:42:08.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:42:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:42:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:42:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-10T16:42:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T16:42:09.200 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' does not exist 2024-09-10T16:42:09.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T16:42:09.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T16:42:09.693 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T16:42:09.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:42:10.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T16:42:12.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-10T16:42:12.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:42:12.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:42:12.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:42:12.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:42:12.674 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:42:12.674 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:42:12.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:42:12.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:42:12.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:42:12.769 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:42:13.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:42:13.155 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:42:13.155 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T16:42:13.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:42:13.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:42:13.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:42:13.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-10T16:42:13.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-10T16:42:13.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-10T16:42:13.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:42:13.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:42:13.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-10T16:42:13.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-10T16:42:13.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2024-09-10T16:42:13.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:42:13.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509447' 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-10T16:42:13.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:42:13.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:42:13.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-10T16:42:13.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:42:13.498 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836502 2024-09-10T16:42:13.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-10T16:42:13.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-10T16:42:13.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:42:13.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-10T16:42:13.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T16:42:14.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T16:42:14.904 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:42:15.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-10T16:42:15.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:42:15.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-10T16:42:15.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:42:15.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:42:15.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-10T16:42:15.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:42:15.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-10T16:42:15.357 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949672980 2024-09-10T16:42:15.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-10T16:42:15.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:42:15.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-10T16:42:15.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:42:15.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-10T16:42:15.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:42:15.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:42:15.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-10T16:42:15.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:42:15.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-10T16:42:15.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-10T16:42:15.747 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509447 2024-09-10T16:42:15.748 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:42:16.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-10T16:42:16.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:42:16.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:42:16.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:42:16.668 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:42:16.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:42:16.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:42:17.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T16:42:17.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:42:17.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:42:17.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:42:17.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T16:42:17.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:42:17.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:42:17.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-10T16:42:17.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-10T16:42:18.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:18.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:18.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:18.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:18.141 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:18.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:18.153 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:17.838+0000 7f30dae88640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.153 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:18.055+0000 7f30dae88640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.153 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:18.055+0000 7f30dae88640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-10T16:42:18.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-10T16:42:18.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:18.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:18.838 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:18.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:18.851 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:18.427+0000 7f30dae88640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.851 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:18.754+0000 7f30dae88640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.851 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:18.754+0000 7f30dae88640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:18.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-10T16:42:18.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:19.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:19.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:19.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:19.472 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:19.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:19.485 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.167+0000 7f30dae88640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:19.485 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.387+0000 7f30dae88640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:19.485 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.387+0000 7f30dae88640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-10T16:42:19.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-10T16:42:19.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:19.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:20.065 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:20.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:20.077 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.759+0000 7f30dae88640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.078 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.981+0000 7f30dae88640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.078 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:19.981+0000 7f30dae88640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-10T16:42:20.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-10T16:42:20.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:20.686 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:20.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:20.698 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:20.381+0000 7f30dae88640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.698 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:20.600+0000 7f30dae88640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.698 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:20.600+0000 7f30dae88640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-10T16:42:20.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-10T16:42:20.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:21.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-10T16:42:21.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:21.449 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:21.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:21.460 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:21.012+0000 7f30dae88640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-10T16:42:21.460 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:21.363+0000 7f30dae88640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-10T16:42:21.460 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:42:21.363+0000 7f30dae88640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-10T16:42:21.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:42:21.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:42:21.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:42:21.569 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:42:21.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:42:21.570 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:42:21.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:42:21.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:42:21.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:42:21.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:21.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:42:21.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:42:21.572 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:21.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:42:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:42:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:42:21.587 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:42:21.587 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:21.587 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:21.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:42:21.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:42:21.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:42:21.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-10T16:42:21.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-10T16:42:21.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:42:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:42:21.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:42:21.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:42:21.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:42:21.592 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:42:21.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:42:21.593 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:21.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:42:21.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:42:21.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:42:21.596 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:42:21.596 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:21.596 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:21.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:42:21.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:42:21.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:42:21.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:42:21.598 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:42:21.598 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:21.599 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:21.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:42:21.600 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:42:21.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:42:21.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:42:21.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-10T16:42:22.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:42:22.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:22.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-10T16:42:22.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:22.250 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:22.251 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:22.251 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:22.251 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:22.251 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:22.252 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:42:22.252 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:42:22.334 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:42:22.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:42:22.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:42:22.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:22.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:42:22.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:42:22.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:42:22.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:42:22.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:42:23.254 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:23.255 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:23.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:42:23.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:42:23.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-10T16:42:23.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:42:23.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:23.354 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:23.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-10T16:42:23.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:23.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:42:23.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:42:23.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:42:23.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:42:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cc3231f3-1c08-4ad9-aa12-85afe22f755e 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cc3231f3-1c08-4ad9-aa12-85afe22f755e' 2024-09-10T16:42:23.368 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 cc3231f3-1c08-4ad9-aa12-85afe22f755e 2024-09-10T16:42:23.369 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:42:23.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDvduBmNJWtFhAAnoK52TpiEWJ6rbqEz3KTEg== 2024-09-10T16:42:23.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDvduBmNJWtFhAAnoK52TpiEWJ6rbqEz3KTEg=="}' 2024-09-10T16:42:23.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cc3231f3-1c08-4ad9-aa12-85afe22f755e -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:42:23.632 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:42:23.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:42:23.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQDvduBmNJWtFhAAnoK52TpiEWJ6rbqEz3KTEg== --osd-uuid cc3231f3-1c08-4ad9-aa12-85afe22f755e 2024-09-10T16:42:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:23.668+0000 7fe0a47d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:23.717 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:23.716+0000 7fe0a47d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:23.718 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:23.717+0000 7fe0a47d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:23.719 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:23.717+0000 7fe0a47d4640 -1 bdev(0x560654db9c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:42:23.719 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:23.718+0000 7fe0a47d4640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:42:29.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:42:29.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:42:29.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:42:29.389 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:42:29.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:42:29.915 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:42:29.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:42:29.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-10T16:42:29.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:42:29.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:42:29.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:42:29.941 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:29.939+0000 7fe4e2c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:29.955 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:29.954+0000 7fe4e2c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:29.957 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:29.955+0000 7fe4e2c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:30.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:42:30.362 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:42:30.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:30.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:30.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:31.296 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:31.295+0000 7fe4e2c1b640 -1 Falling back to public interface 2024-09-10T16:42:31.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:31.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:31.779 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:42:31.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:42:31.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:31.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:32.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:32.891 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:32.890+0000 7fe4e2c1b640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:42:33.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:33.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:33.206 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:42:33.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:42:33.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:33.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:33.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:34.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:34.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:34.599 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:42:34.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:42:34.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:34.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:35.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:36.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:36.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:36.023 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:42:36.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:42:36.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:36.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:36.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:37.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:37.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:37.449 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:42:37.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:42:37.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:37.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:37.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:38.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:38.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:38.898 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:42:38.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:42:38.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:38.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:39.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:40.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:40.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:40.316 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:42:40.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:42:40.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:40.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:40.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:41.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:41.789 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:42:41.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:41.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:42:41.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:41.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:42.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:43.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:43.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:43.191 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:42:43.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:42:43.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:43.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:43.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:44.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:44.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:44.844 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:42:44.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:42:44.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:44.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:45.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:46.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:46.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:46.589 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:42:46.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:42:46.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:46.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:47.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:48.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:48.367 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:42:48.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:48.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:42:48.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:48.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:49.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:50.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:50.292 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:42:50.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:50.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:42:50.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:50.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:51.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:52.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:52.041 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:42:52.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:52.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:42:52.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:52.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:52.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:53.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:53.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:53.831 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:42:53.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:42:53.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:53.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:54.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:42:55.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:42:55.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:42:55.796 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:42:55.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:42:55.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:42:55.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:42:56.264 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1086128263,v1:127.0.0.1:6803/1086128263] [v2:127.0.0.1:6804/1086128263,v1:127.0.0.1:6805/1086128263] exists,up cc3231f3-1c08-4ad9-aa12-85afe22f755e 2024-09-10T16:42:56.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:42:56.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:42:56.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:42:56.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-10T16:42:56.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:42:56.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:42:56.345 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:56.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:42:56.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:42:56.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:42:56.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:42:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:42:56.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:42:56.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:42:56.464 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:42:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:42:56.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:42:56.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:42:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:42:56.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:42:56.466 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:56.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:42:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:42:56.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:42:56.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:42:56.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:56.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:42:56.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:42:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:42:56.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:42:56.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:42:56.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:42:56.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:42:56.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:42:56.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:42:56.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:42:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:42:56.480 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:42:56.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:42:56.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:42:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:42:56.483 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:42:56.483 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:56.483 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:42:56.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:42:56.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:42:56.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:42:56.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:42:56.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:56.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:56.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:42:56.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:42:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:42:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:42:56.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:56.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:56.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-10T16:42:57.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:42:57.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:57.112 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:57.113 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:42:57.113 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:42:57.190 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:42:57.191 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:42:57.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:42:57.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:57.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:42:57.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:42:57.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:42:57.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:42:57.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:42:58.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:42:58.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:42:58.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:42:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:42:58.233 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:42:58.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:42:58.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:42:58.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:42:58.246 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:42:58.247 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 0c42bd39-1edb-4163-9159-ba95c2be127f 2024-09-10T16:42:58.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c42bd39-1edb-4163-9159-ba95c2be127f 2024-09-10T16:42:58.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0c42bd39-1edb-4163-9159-ba95c2be127f' 2024-09-10T16:42:58.247 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:42:58.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASd+BmT/F4DxAAStGFf/qjkTSfjKLIF+7YDQ== 2024-09-10T16:42:58.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASd+BmT/F4DxAAStGFf/qjkTSfjKLIF+7YDQ=="}' 2024-09-10T16:42:58.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c42bd39-1edb-4163-9159-ba95c2be127f -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:42:58.518 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:42:58.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:42:58.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQASd+BmT/F4DxAAStGFf/qjkTSfjKLIF+7YDQ== --osd-uuid 0c42bd39-1edb-4163-9159-ba95c2be127f 2024-09-10T16:42:58.557 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:58.555+0000 7f8ffb411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:58.587 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:58.586+0000 7f8ffb411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:58.588 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:58.587+0000 7f8ffb411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:42:58.589 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:58.587+0000 7f8ffb411640 -1 bdev(0x564a5d70fc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:42:58.589 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:42:58.587+0000 7f8ffb411640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:43:04.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:43:04.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:43:04.083 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:43:04.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:43:04.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:43:04.619 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:43:04.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:43:04.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-10T16:43:04.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:43:04.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:43:04.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:43:04.644 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:04.643+0000 7f33ea213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:04.659 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:04.657+0000 7f33ea213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:04.660 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:04.658+0000 7f33ea213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:05.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:43:05.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:43:05.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:43:05.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:43:05.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:43:05.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:05.140 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:43:05.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:43:05.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:05.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:05.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:06.009+0000 7f33ea213640 -1 Falling back to public interface 2024-09-10T16:43:06.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:06.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:06.575 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:43:06.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:43:06.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:06.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:06.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:07.611 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:07.609+0000 7f33ea213640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:43:08.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:08.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:08.001 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:43:08.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:43:08.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:08.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:08.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:09.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:09.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:09.525 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:43:09.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:43:09.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:09.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:09.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:10.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:10.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:10.928 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:43:10.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:43:10.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:10.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:11.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:12.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:12.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:12.360 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:43:12.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:43:12.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:12.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:12.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:13.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:13.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:13.751 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:43:13.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:43:13.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:13.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:14.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:15.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:15.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:15.169 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:43:15.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:43:15.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:15.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:15.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:16.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:16.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:16.659 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:43:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:43:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:17.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:18.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:18.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:18.063 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:43:18.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:43:18.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:18.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:18.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:19.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:19.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:19.492 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:43:19.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:43:19.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:19.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:20.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:21.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:21.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:21.340 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:43:21.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:43:21.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:21.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:21.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:22.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:22.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:22.919 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:43:22.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:43:22.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:22.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:23.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:24.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:24.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:24.552 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:43:24.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:43:24.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:24.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:25.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:26.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:26.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:26.700 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:43:26.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:43:26.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:26.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:27.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:28.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:28.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:28.725 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:43:28.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:43:28.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:28.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:29.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:30.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:30.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:30.563 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:43:30.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:43:30.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:30.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:30.983 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1494179609,v1:127.0.0.1:6803/1494179609] [v2:127.0.0.1:6804/1494179609,v1:127.0.0.1:6805/1494179609] exists,up 0c42bd39-1edb-4163-9159-ba95c2be127f 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:43:30.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:43:30.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:43:31.064 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:43:31.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:43:31.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:43:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:43:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:43:31.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:43:31.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:43:31.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:43:31.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:43:31.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:43:31.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:43:31.184 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:43:31.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:43:31.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:43:31.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:43:31.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:43:31.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:43:31.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:43:31.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:43:31.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:43:31.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:43:31.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:43:31.196 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:43:31.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:43:31.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:43:31.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:43:31.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:43:31.197 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:43:31.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:43:31.197 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:43:31.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:43:31.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:43:31.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:43:31.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:43:31.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:43:31.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:43:31.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:43:31.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:43:31.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:43:31.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:43:31.202 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:43:31.202 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.202 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:43:31.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:43:31.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:43:31.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:43:31.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:43:31.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:43:31.206 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:43:31.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:43:31.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:43:31.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-10T16:43:31.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:43:31.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:43:31.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:43:31.663 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:43:31.663 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.663 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.663 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:43:31.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-10T16:43:31.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:43:31.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:43:31.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:43:31.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:43:31.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:43:31.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:43:31.857 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:43:31.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:31.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:43:31.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:43:31.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:43:32.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:43:32.851 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:43:32.851 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:43:32.851 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:43:32.851 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:43:32.851 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:32.852 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:32.852 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:43:32.852 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:43:32.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:43:32.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-10T16:43:32.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:43:32.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:43:32.880 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:43:32.880 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:32.881 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:32.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:43:32.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:43:32.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:43:32.952 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:43:32.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:43:32.964 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-10T16:43:32.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:43:32.966 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:43:32.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cf3191f3-4a99-4b02-8766-2adf0e9a3162 2024-09-10T16:43:32.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cf3191f3-4a99-4b02-8766-2adf0e9a3162' 2024-09-10T16:43:32.966 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 cf3191f3-4a99-4b02-8766-2adf0e9a3162 2024-09-10T16:43:32.967 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:43:32.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0d+BmNa1bOhAADnqmZlCUcYkZ+R18PydSfA== 2024-09-10T16:43:32.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0d+BmNa1bOhAADnqmZlCUcYkZ+R18PydSfA=="}' 2024-09-10T16:43:32.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cf3191f3-4a99-4b02-8766-2adf0e9a3162 -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:43:33.254 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:43:33.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:43:33.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQA0d+BmNa1bOhAADnqmZlCUcYkZ+R18PydSfA== --osd-uuid cf3191f3-4a99-4b02-8766-2adf0e9a3162 2024-09-10T16:43:33.293 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:33.291+0000 7f7ab9e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:33.323 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:33.322+0000 7f7ab9e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:33.324 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:33.323+0000 7f7ab9e44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:33.324 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:33.323+0000 7f7ab9e44640 -1 bdev(0x55b08c0d9c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:43:33.325 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:33.323+0000 7f7ab9e44640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:43:38.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:43:38.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:43:38.764 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:43:38.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:43:38.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:43:39.295 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:43:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:43:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-10T16:43:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:43:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:43:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:43:39.320 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:39.319+0000 7f1a5ae9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:39.322 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:39.321+0000 7f1a5ae9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:39.324 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:39.322+0000 7f1a5ae9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:43:39.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:43:39.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:43:39.715 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:43:39.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:39.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:40.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:40.660 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:40.659+0000 7f1a5ae9c640 -1 Falling back to public interface 2024-09-10T16:43:41.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:41.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:41.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:43:41.142 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:43:41.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:41.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:41.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:42.487 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:43:42.485+0000 7f1a5ae9c640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:43:42.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:42.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:42.560 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:43:42.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:43:42.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:42.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:43.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:44.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:44.082 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:43:44.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:43:44.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:44.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:44.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:45.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:45.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:45.530 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:43:45.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:43:45.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:45.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:45.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:46.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:46.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:46.941 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:43:46.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:43:46.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:46.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:47.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:48.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:48.363 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:43:48.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:48.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:43:48.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:48.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:48.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:49.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:49.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:49.754 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:43:49.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:43:49.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:49.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:50.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:51.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:51.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:51.218 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:43:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:43:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:51.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:52.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:52.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:52.611 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:43:52.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:43:52.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:52.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:53.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:54.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:54.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:54.024 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:43:54.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:43:54.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:54.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:54.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:55.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:55.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:55.863 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:43:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:43:55.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:55.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:56.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:57.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:57.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:57.446 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:43:57.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:43:57.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:57.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:58.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:43:59.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:43:59.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:43:59.092 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:43:59.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:43:59.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:43:59.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:43:59.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:00.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:00.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:00.986 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:44:00.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:44:00.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:00.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:01.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:02.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:02.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:02.416 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:44:02.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:44:02.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:02.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:03.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:04.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:04.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:04.109 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:44:04.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:44:04.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:04.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:05.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:05.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:05.887 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:44:05.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:44:05.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:05.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:06.276 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1267376887,v1:127.0.0.1:6803/1267376887] [v2:127.0.0.1:6804/1267376887,v1:127.0.0.1:6805/1267376887] exists,up cf3191f3-4a99-4b02-8766-2adf0e9a3162 2024-09-10T16:44:06.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:44:06.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:44:06.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:44:06.279 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-10T16:44:06.279 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:44:06.279 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:44:06.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:44:06.357 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:44:06.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:44:06.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:44:06.369 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:44:06.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:44:06.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:44:06.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:44:06.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:44:06.475 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:44:06.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:44:06.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:44:06.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:44:06.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:44:06.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:44:06.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:06.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:44:06.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:44:06.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:06.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:44:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:44:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:44:06.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:44:06.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:06.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:06.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:44:06.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:44:06.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:44:06.490 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:44:06.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:44:06.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:44:06.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:44:06.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:44:06.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:44:06.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:06.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:44:06.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:44:06.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:06.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:44:06.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:44:06.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:44:06.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:44:06.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:06.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:06.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:44:06.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:44:06.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:44:06.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:44:06.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:44:06.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:06.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:06.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-10T16:44:06.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:44:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:44:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:44:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:44:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-10T16:44:06.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:06.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-10T16:44:07.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:44:07.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:44:07.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:44:07.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:44:07.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:07.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:44:07.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:44:07.244 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:44:07.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:44:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:08.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:44:08.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:44:08.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-10T16:44:08.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:44:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:44:08.279 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:08.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-10T16:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:44:08.292 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:44:08.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=87ad943a-e0eb-45ac-994d-a7ef08d8890a 2024-09-10T16:44:08.292 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 87ad943a-e0eb-45ac-994d-a7ef08d8890a 2024-09-10T16:44:08.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 87ad943a-e0eb-45ac-994d-a7ef08d8890a' 2024-09-10T16:44:08.293 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:44:08.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBYd+BmWHpPEhAA6SrdzlXC8c0q0MPBnInD6g== 2024-09-10T16:44:08.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBYd+BmWHpPEhAA6SrdzlXC8c0q0MPBnInD6g=="}' 2024-09-10T16:44:08.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 87ad943a-e0eb-45ac-994d-a7ef08d8890a -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:44:08.562 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:44:08.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:44:08.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQBYd+BmWHpPEhAA6SrdzlXC8c0q0MPBnInD6g== --osd-uuid 87ad943a-e0eb-45ac-994d-a7ef08d8890a 2024-09-10T16:44:08.601 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:08.600+0000 7fed0f134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:08.631 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:08.629+0000 7fed0f134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:08.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:08.630+0000 7fed0f134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:08.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:08.631+0000 7fed0f134640 -1 bdev(0x560f7dd27c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:44:08.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:08.631+0000 7fed0f134640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:44:14.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:44:14.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:44:14.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:44:14.419 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:44:14.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:44:14.954 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:44:14.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:44:14.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-10T16:44:14.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:44:14.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:44:14.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:44:14.981 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:14.979+0000 7fe2ae3d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:14.992 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:14.990+0000 7fe2ae3d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:14.993 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:14.992+0000 7fe2ae3d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:15.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:44:15.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:44:15.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:44:15.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:44:15.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:44:15.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:15.415 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:44:15.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:44:15.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:15.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:15.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:16.585 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:16.584+0000 7fe2ae3d9640 -1 Falling back to public interface 2024-09-10T16:44:16.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:16.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:16.797 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:44:16.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:44:16.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:16.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:17.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:18.128 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:18.127+0000 7fe2ae3d9640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:44:18.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:18.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:18.217 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:44:18.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:44:18.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:18.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:18.738 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:19.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:19.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:19.739 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:44:19.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:44:19.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:19.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:20.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:21.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:21.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:21.188 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:44:21.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:44:21.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:21.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:21.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:22.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:22.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:22.593 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:44:22.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:44:22.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:22.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:23.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:24.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:24.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:24.032 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:44:24.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:44:24.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:24.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:24.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:25.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:25.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:25.447 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:44:25.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:44:25.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:25.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:25.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:26.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:26.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:26.862 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:44:26.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:44:26.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:26.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:27.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:28.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:28.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:28.282 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:44:28.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:44:28.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:28.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:28.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:29.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:29.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:29.672 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:44:29.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:44:29.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:29.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:30.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:31.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:31.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:31.358 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:44:31.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:44:31.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:31.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:31.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:32.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:32.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:32.827 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:44:32.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:44:32.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:32.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:33.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:34.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:34.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:34.676 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:44:34.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:44:34.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:34.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:35.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:36.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:36.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:36.252 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:44:36.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:44:36.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:36.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:37.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:38.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:38.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:38.323 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:44:38.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:44:38.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:38.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:38.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:39.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:39.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:39.722 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:44:39.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:44:39.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:39.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:40.322 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/201416580,v1:127.0.0.1:6803/201416580] [v2:127.0.0.1:6804/201416580,v1:127.0.0.1:6805/201416580] exists,up 87ad943a-e0eb-45ac-994d-a7ef08d8890a 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:40.323 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:40.324 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:40.324 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:44:40.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-10T16:44:40.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:44:40.402 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:44:40.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:44:40.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:44:40.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:44:40.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:40.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:44:40.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:44:40.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:44:40.414 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:44:40.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:44:40.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:44:40.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:44:40.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:44:40.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:44:40.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:44:40.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:44:40.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:44:40.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:44:40.521 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:44:40.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:40.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:44:40.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:44:40.522 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:40.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:44:40.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:44:40.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:44:40.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:44:40.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:40.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:40.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:40.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:44:40.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:44:40.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:44:40.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:44:40.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:44:40.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:44:40.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:44:40.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:40.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:40.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:44:40.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:44:40.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:44:40.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:44:40.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:44:40.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:40.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:40.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:44:40.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:44:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:44:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-10T16:44:41.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:44:41.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:41.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T16:44:41.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:41.188 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:44:41.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:44:41.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:44:41.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:44:41.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:44:41.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:44:41.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:44:41.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-10T16:44:41.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:44:41.416 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:44:41.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:44:41.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:41.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:44:41.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:44:41.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:44:41.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:44:41.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:44:42.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:44:42.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:42.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:42.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:42.267 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:42.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:42.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:44:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:44:42.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-10T16:44:42.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:44:42.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:44:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:44:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:44:42.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0b2b8926-e486-413a-be03-90fe1d35441b 2024-09-10T16:44:42.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0b2b8926-e486-413a-be03-90fe1d35441b' 2024-09-10T16:44:42.300 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 0b2b8926-e486-413a-be03-90fe1d35441b 2024-09-10T16:44:42.300 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:44:42.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6d+Bmu0ymEhAAyfiBCasQO18yFAYx8Zkt5w== 2024-09-10T16:44:42.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6d+Bmu0ymEhAAyfiBCasQO18yFAYx8Zkt5w=="}' 2024-09-10T16:44:42.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0b2b8926-e486-413a-be03-90fe1d35441b -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:44:42.586 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:44:42.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:44:42.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6d+Bmu0ymEhAAyfiBCasQO18yFAYx8Zkt5w== --osd-uuid 0b2b8926-e486-413a-be03-90fe1d35441b 2024-09-10T16:44:42.623 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:42.622+0000 7f443cb23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:42.654 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:42.653+0000 7f443cb23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:42.655 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:42.654+0000 7f443cb23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:42.656 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:42.654+0000 7f443cb23640 -1 bdev(0x563ccb19dc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:44:42.656 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:42.654+0000 7f443cb23640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:44:48.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:44:48.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:44:48.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:44:48.405 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:44:48.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:44:48.924 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:44:48.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:44:48.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:44:48.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:44:48.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:44:48.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:44:48.950 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:48.948+0000 7f40745c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:48.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:48.981+0000 7f40745c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:48.984 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:48.982+0000 7f40745c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:44:49.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:44:49.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:44:49.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:49.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:49.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:50.330 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:50.329+0000 7f40745c1640 -1 Falling back to public interface 2024-09-10T16:44:50.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:50.765 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:44:50.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:50.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:44:50.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:50.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:51.917 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:44:51.916+0000 7f40745c1640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:44:52.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:52.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:52.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:44:52.195 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:44:52.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:52.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:52.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:53.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:53.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:53.665 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:44:53.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:44:53.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:53.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:54.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:55.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:55.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:55.075 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:44:55.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:44:55.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:55.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:55.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:56.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:44:56.483 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:44:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:56.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:56.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:57.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:57.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:57.880 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:44:57.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:44:57.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:57.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:58.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:44:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:44:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:44:59.283 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:44:59.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:44:59.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:44:59.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:44:59.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:00.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:00.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:00.801 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:45:00.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:45:00.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:00.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:01.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:02.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:02.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:02.192 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:45:02.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:45:02.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:02.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:02.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:03.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:03.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:03.689 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:45:03.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:45:03.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:03.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:04.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:05.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:05.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:05.386 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:45:05.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:45:05.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:05.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:06.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:06.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:06.965 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:45:06.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:45:06.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:06.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:07.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:08.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:08.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:08.653 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:45:08.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:45:08.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:08.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:09.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:10.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:10.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:10.318 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:45:10.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:45:10.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:10.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:10.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:11.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:11.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:11.876 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:45:11.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:45:11.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:11.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:13.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:13.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:13.598 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:45:13.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:45:13.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:13.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:14.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:15.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:15.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:15.507 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:45:15.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:45:15.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:15.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:15.924 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/104675744,v1:127.0.0.1:6803/104675744] [v2:127.0.0.1:6804/104675744,v1:127.0.0.1:6805/104675744] exists,up 0b2b8926-e486-413a-be03-90fe1d35441b 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:45:15.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-10T16:45:15.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:45:16.000 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:45:16.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:45:16.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:45:16.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:45:16.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:45:16.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:45:16.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:45:16.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:45:16.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:45:16.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:45:16.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:45:16.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:16.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:45:16.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:45:16.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:16.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:45:16.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:45:16.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:45:16.129 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:45:16.129 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.129 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:45:16.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:45:16.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:45:16.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:45:16.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:45:16.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:45:16.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:16.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:45:16.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:45:16.136 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:16.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:45:16.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:45:16.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:45:16.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:45:16.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.139 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:45:16.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:45:16.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:45:16.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:45:16.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:45:16.141 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.141 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:45:16.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:45:16.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:16.650 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:45:16.650 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:16.650 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:16.651 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:16.651 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.651 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:16.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:16.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.837 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.837 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:16.837 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:45:16.837 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:45:16.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:45:16.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:45:16.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:45:16.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:45:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:45:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:45:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:45:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:17.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:17.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:45:17.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:45:17.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:45:17.938 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:45:17.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:17.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:45:17.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:17.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:45:17.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:45:17.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6aa37555-d8e4-421a-9fda-467c60d84545 2024-09-10T16:45:17.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6aa37555-d8e4-421a-9fda-467c60d84545' 2024-09-10T16:45:17.953 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 6aa37555-d8e4-421a-9fda-467c60d84545 2024-09-10T16:45:17.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:45:17.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdd+BmoTKKORAAHBbMzfjQPZTQlO/Bgz08rw== 2024-09-10T16:45:17.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdd+BmoTKKORAAHBbMzfjQPZTQlO/Bgz08rw=="}' 2024-09-10T16:45:17.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6aa37555-d8e4-421a-9fda-467c60d84545 -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:45:18.217 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:45:18.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:45:18.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQCdd+BmoTKKORAAHBbMzfjQPZTQlO/Bgz08rw== --osd-uuid 6aa37555-d8e4-421a-9fda-467c60d84545 2024-09-10T16:45:18.255 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:18.253+0000 7f3d1361e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:18.285 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:18.284+0000 7f3d1361e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:18.286 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:18.285+0000 7f3d1361e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:18.287 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:18.285+0000 7f3d1361e640 -1 bdev(0x55aaa371fc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:45:18.287 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:18.285+0000 7f3d1361e640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:45:23.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:45:23.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:45:23.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:45:23.925 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:45:23.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:45:24.453 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:45:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:45:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-10T16:45:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:45:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:45:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:45:24.479 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:24.477+0000 7f75c5aeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:24.504 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:24.503+0000 7f75c5aeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:24.505 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:24.504+0000 7f75c5aeb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:24.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:45:24.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:45:24.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:45:24.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:45:24.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:45:24.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:24.877 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:45:24.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:45:24.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:24.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:25.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:25.604 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:25.603+0000 7f75c5aeb640 -1 Falling back to public interface 2024-09-10T16:45:26.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:26.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:26.295 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:45:26.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:45:26.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:26.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:26.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:27.664 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:27.662+0000 7f75c5aeb640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:45:27.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:27.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:27.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:45:27.722 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:45:27.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:27.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:28.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:29.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:29.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:45:29.326 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:45:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:29.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:30.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:30.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:30.794 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:45:30.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:45:30.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:30.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:31.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:32.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:32.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:32.218 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:45:32.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:45:32.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:32.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:32.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:33.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:33.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:33.632 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:45:33.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:45:33.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:33.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:34.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:35.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:35.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:35.018 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:45:35.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:45:35.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:35.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:35.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:36.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:36.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:36.495 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:45:36.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:45:36.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:36.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:36.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:37.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:37.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:37.889 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:45:37.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:45:37.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:37.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:38.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:39.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:39.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:39.295 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:45:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:45:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:40.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:41.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:41.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:41.241 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:45:41.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:45:41.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:41.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:41.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:42.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:42.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:42.861 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:45:42.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:45:42.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:42.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:43.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:44.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:44.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:44.537 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:45:44.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:45:44.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:44.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:45.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:46.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:46.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:46.110 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:45:46.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:45:46.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:46.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:47.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:48.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:48.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:48.310 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:45:48.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:45:48.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:48.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:49.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:45:50.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:45:50.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:50.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:45:50.113 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:45:50.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:50.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:45:50.532 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2795610390,v1:127.0.0.1:6803/2795610390] [v2:127.0.0.1:6804/2795610390,v1:127.0.0.1:6805/2795610390] exists,up 6aa37555-d8e4-421a-9fda-467c60d84545 2024-09-10T16:45:50.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:45:50.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:45:50.612 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:45:50.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:45:50.621 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:45:16.679+0000 7f390f71ec80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:45:50.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:45:50.623 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:45:25.604+0000 7f75c5aeb640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:45:50.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:45:50.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:45:50.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:45:50.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:45:50.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:45:50.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:45:50.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:45:50.732 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:45:50.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:50.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:45:50.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:45:50.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:50.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:45:50.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:45:50.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:45:50.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:45:50.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:50.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:50.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:45:50.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:45:50.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:45:50.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:45:50.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:45:50.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:45:50.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:50.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:45:50.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:45:50.749 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:45:50.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:45:50.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:45:50.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:45:50.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:45:50.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:45:50.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:45:50.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:50.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:50.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:45:50.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:45:50.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:45:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:51.219 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:51.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:51.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:51.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:51.413 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:51.414 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:45:51.414 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:45:51.494 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:45:51.495 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:45:51.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:45:51.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:51.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:45:51.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:45:51.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:45:51.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:45:51.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:52.410 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:52.411 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:45:52.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:45:52.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:45:52.533 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:45:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:45:52.545 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:45:52.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d 2024-09-10T16:45:52.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d' 2024-09-10T16:45:52.548 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d 2024-09-10T16:45:52.548 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:45:52.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDAd+BmO1ppIRAA1jXWAWkovDZQnTuhKVmcow== 2024-09-10T16:45:52.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDAd+BmO1ppIRAA1jXWAWkovDZQnTuhKVmcow=="}' 2024-09-10T16:45:52.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:45:52.821 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:45:52.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:45:52.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQDAd+BmO1ppIRAA1jXWAWkovDZQnTuhKVmcow== --osd-uuid fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d 2024-09-10T16:45:52.860 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:52.858+0000 7fa536f3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:52.890 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:52.888+0000 7fa536f3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:52.890 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:52.889+0000 7fa536f3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:52.891 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:52.890+0000 7fa536f3e640 -1 bdev(0x5585416cbc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:45:52.891 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:52.890+0000 7fa536f3e640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:45:58.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:45:58.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:45:58.653 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:45:58.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:45:58.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:45:59.183 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:45:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:45:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-10T16:45:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:45:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:45:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:45:59.209 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:59.207+0000 7f0d56745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:59.225 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:59.223+0000 7f0d56745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:59.226 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:45:59.225+0000 7f0d56745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:45:59.647 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:45:59.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:45:59.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:45:59.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:00.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:00.586 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:00.585+0000 7f0d56745640 -1 Falling back to public interface 2024-09-10T16:46:01.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:01.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:01.022 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:46:01.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:46:01.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:01.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:01.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:02.185 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:02.183+0000 7f0d56745640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:46:02.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:02.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:02.417 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:46:02.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:46:02.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:02.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:02.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:03.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:03.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:03.929 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:46:03.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:46:03.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:03.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:05.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:05.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:05.348 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:46:05.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:46:05.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:05.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:05.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:06.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:06.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:06.748 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:46:06.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:46:06.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:06.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:07.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:08.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:08.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:08.223 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:46:08.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:46:08.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:08.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:08.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:09.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:09.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:09.614 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:46:09.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:46:09.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:09.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:10.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:11.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:11.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:11.081 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:46:11.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:46:11.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:11.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:11.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:12.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:12.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:12.470 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:46:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:46:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:12.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:13.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:13.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:13.877 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:46:13.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:46:13.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:13.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:14.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:15.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:15.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:15.779 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:46:15.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:46:15.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:15.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:16.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:17.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:17.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:17.249 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:46:17.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:46:17.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:17.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:17.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:18.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:18.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:18.977 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:46:18.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:46:18.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:18.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:19.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:20.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:20.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:20.938 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:46:20.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:46:20.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:20.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:21.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:22.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:22.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:22.896 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:46:22.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:46:22.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:22.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:23.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:24.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:24.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:24.792 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:46:24.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:46:24.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:24.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:25.221 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/530795233,v1:127.0.0.1:6803/530795233] [v2:127.0.0.1:6804/530795233,v1:127.0.0.1:6805/530795233] exists,up fad0ed07-9f2e-47dc-a402-50b1e3fa0f9d 2024-09-10T16:46:25.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:46:25.224 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:25.225 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:25.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:46:25.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:46:25.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:46:25.303 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:46:25.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:46:25.313 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:45:51.248+0000 7f8fcec1dc80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:46:25.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:46:25.314 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:46:00.586+0000 7f0d56745640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:46:25.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:46:25.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:25.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:46:25.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:46:25.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:46:25.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:46:25.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:46:25.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:46:25.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:46:25.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:46:25.421 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:46:25.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:46:25.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:46:25.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:46:25.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:46:25.423 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:46:25.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:46:25.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:46:25.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:46:25.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:46:25.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:46:25.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:46:25.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:46:25.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:46:25.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:25.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:25.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:46:25.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:46:25.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:46:25.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:46:25.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:46:25.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:46:25.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:46:25.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:46:25.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:46:25.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:46:25.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:46:25.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:46:25.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:46:25.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:46:25.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:46:25.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:46:25.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:46:25.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:46:25.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:46:25.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:25.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:25.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:46:25.444 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:46:25.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:46:25.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:26.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:46:26.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:46:26.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:46:26.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:46:26.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:46:26.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:46:26.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:46:26.164 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:46:26.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:46:26.166 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:46:26.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:46:26.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:26.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:46:26.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:46:26.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:46:26.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:46:26.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:46:27.088 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:46:27.088 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:46:27.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:46:27.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:46:27.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:46:27.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:46:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:46:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:27.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:46:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:46:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:46:27.187 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:46:27.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:27.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:46:27.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:46:27.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:46:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e7e7135-bee8-46f9-8321-2b0adec93551 2024-09-10T16:46:27.200 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 0e7e7135-bee8-46f9-8321-2b0adec93551 2024-09-10T16:46:27.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0e7e7135-bee8-46f9-8321-2b0adec93551' 2024-09-10T16:46:27.201 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:46:27.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjd+BmO6HLDBAAqSphBDclztlBBrXME+H/zw== 2024-09-10T16:46:27.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjd+BmO6HLDBAAqSphBDclztlBBrXME+H/zw=="}' 2024-09-10T16:46:27.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e7e7135-bee8-46f9-8321-2b0adec93551 -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:46:27.475 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:46:27.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:46:27.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQDjd+BmO6HLDBAAqSphBDclztlBBrXME+H/zw== --osd-uuid 0e7e7135-bee8-46f9-8321-2b0adec93551 2024-09-10T16:46:27.513 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:27.512+0000 7f49e72fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:27.535 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:27.534+0000 7f49e72fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:27.536 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:27.535+0000 7f49e72fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:27.536 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:27.535+0000 7f49e72fe640 -1 bdev(0x55a3331f1c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:46:27.537 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:27.535+0000 7f49e72fe640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:46:33.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:46:33.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:46:33.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:46:33.231 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:46:33.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:46:33.759 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:46:33.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:46:33.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-10T16:46:33.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:46:33.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:46:33.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:46:33.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:33.783+0000 7efe1fdfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:33.804 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:33.802+0000 7efe1fdfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:33.805 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:33.804+0000 7efe1fdfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:46:34.209 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:46:34.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:34.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:46:34.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:34.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:34.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:34.896 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:34.895+0000 7efe1fdfd640 -1 Falling back to public interface 2024-09-10T16:46:35.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:46:35.628 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:46:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:35.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:36.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:36.605 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:46:36.604+0000 7efe1fdfd640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:46:37.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:37.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:37.071 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:46:37.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:46:37.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:37.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:37.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:38.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:38.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:38.567 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:46:38.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:46:38.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:38.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:38.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:39.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:39.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:39.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:46:39.979 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:46:39.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:39.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:40.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:41.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:41.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:41.404 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:46:41.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:46:41.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:41.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:41.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:42.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:42.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:42.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:46:42.825 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:46:42.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:42.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:43.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:44.406 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:46:44.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:46:44.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:44.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:44.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:45.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:45.803 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:46:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:46:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:46.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:47.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:47.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:47.195 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:46:47.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:46:47.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:47.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:47.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:48.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:48.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:48.652 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:46:48.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:46:48.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:48.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:49.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:50.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:50.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:50.589 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:46:50.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:46:50.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:50.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:51.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:52.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:52.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:52.241 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:46:52.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:46:52.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:52.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:52.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:53.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:53.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:53.884 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:46:53.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:46:53.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:53.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:54.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:55.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:55.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:55.703 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:46:55.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:46:55.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:55.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:56.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:57.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:46:57.909 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:46:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:46:58.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:46:59.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:46:59.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:46:59.636 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:46:59.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:46:59.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:46:59.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:00.165 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1759583039,v1:127.0.0.1:6803/1759583039] [v2:127.0.0.1:6804/1759583039,v1:127.0.0.1:6805/1759583039] exists,up 0e7e7135-bee8-46f9-8321-2b0adec93551 2024-09-10T16:47:00.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:47:00.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:47:00.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:47:00.166 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:47:00.166 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:47:00.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:47:00.244 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:47:00.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:47:00.253 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:46:25.930+0000 7f73f7d32c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:47:00.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:47:00.255 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:46:34.896+0000 7efe1fdfd640 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:47:00.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:47:00.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:47:00.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:47:00.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:47:00.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:47:00.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:47:00.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:47:00.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:47:00.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:00.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:47:00.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:47:00.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:00.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:47:00.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:47:00.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:47:00.372 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:47:00.372 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.372 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:47:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:47:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:47:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:47:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:47:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:47:00.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:47:00.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:47:00.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:47:00.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:47:00.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:47:00.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:47:00.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:47:00.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:47:00.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:00.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:47:00.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:47:00.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:00.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:47:00.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:47:00.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:47:00.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:47:00.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:47:00.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:47:00.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:47:00.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:47:00.384 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:47:00.384 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.384 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:47:00.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:47:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:47:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:47:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:47:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:47:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:00.817 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:00.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:00.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:47:00.993 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:47:00.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:47:00.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:47:00.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:47:00.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:47:00.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:47:00.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:00.995 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:00.995 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:00.996 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:00.996 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:00.996 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:00.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:47:00.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:47:01.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:47:01.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:47:01.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:47:01.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:01.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:01.076 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:01.076 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:47:01.077 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:47:01.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:47:01.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:47:01.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:01.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:47:01.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:47:02.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:47:02.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:02.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:47:02.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:47:02.089 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:47:02.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:47:02.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:02.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:47:02.102 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:47:02.103 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 a4a71371-282c-464d-86e0-5328752946ea 2024-09-10T16:47:02.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a4a71371-282c-464d-86e0-5328752946ea 2024-09-10T16:47:02.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a4a71371-282c-464d-86e0-5328752946ea' 2024-09-10T16:47:02.103 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:47:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGeOBmmzfgBhAAYvFE54myBLk/XWHbzRCPbQ== 2024-09-10T16:47:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGeOBmmzfgBhAAYvFE54myBLk/XWHbzRCPbQ=="}' 2024-09-10T16:47:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a4a71371-282c-464d-86e0-5328752946ea -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:47:02.376 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:47:02.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:47:02.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQAGeOBmmzfgBhAAYvFE54myBLk/XWHbzRCPbQ== --osd-uuid a4a71371-282c-464d-86e0-5328752946ea 2024-09-10T16:47:02.414 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:02.413+0000 7fc3df226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:02.445 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:02.443+0000 7fc3df226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:02.446 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:02.445+0000 7fc3df226640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:02.446 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:02.445+0000 7fc3df226640 -1 bdev(0x5556b03c9c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:47:02.446 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:02.445+0000 7fc3df226640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:47:07.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:47:07.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:47:07.994 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:47:07.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:47:07.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:47:08.525 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:47:08.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:47:08.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-10T16:47:08.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:47:08.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:47:08.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:47:08.551 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:08.549+0000 7f306780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:08.585 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:08.583+0000 7f306780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:08.586 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:08.584+0000 7f306780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:08.940 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:47:08.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:47:08.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:08.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:09.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:09.680 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:09.679+0000 7f306780d640 -1 Falling back to public interface 2024-09-10T16:47:10.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:10.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:10.323 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:47:10.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:47:10.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:10.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:10.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:11.345 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:11.344+0000 7f306780d640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:47:11.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:11.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:11.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:47:11.716 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:47:11.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:11.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:12.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:13.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:13.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:13.334 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:47:13.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:47:13.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:13.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:13.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:14.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:14.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:14.730 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:47:14.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:47:14.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:14.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:15.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:16.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:16.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:16.162 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:47:16.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:47:16.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:16.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:16.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:17.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:17.561 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:47:17.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:17.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:47:17.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:17.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:18.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:19.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:19.116 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:47:19.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:19.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:47:19.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:19.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:19.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:20.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:20.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:20.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:47:20.510 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:47:20.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:20.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:20.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:21.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:21.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:21.931 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:47:21.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:47:21.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:21.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:22.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:23.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:23.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:23.549 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:47:23.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:47:23.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:23.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:23.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:24.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:24.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:24.924 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:47:24.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:47:24.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:24.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:26.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:26.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:26.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:47:26.522 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:47:26.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:26.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:27.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:28.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:28.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:28.212 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:47:28.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:47:28.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:28.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:28.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:29.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:29.786 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:47:29.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:29.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:47:29.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:29.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:30.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:31.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:31.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:31.424 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:47:31.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:47:31.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:31.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:32.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:33.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:33.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:33.195 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:47:33.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:47:33.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:33.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:33.613 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1421085598,v1:127.0.0.1:6803/1421085598] [v2:127.0.0.1:6804/1421085598,v1:127.0.0.1:6805/1421085598] exists,up a4a71371-282c-464d-86e0-5328752946ea 2024-09-10T16:47:33.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:47:33.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:47:33.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:47:33.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:47:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:47:33.692 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:47:33.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:47:33.702 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:47:00.844+0000 7f2124f22c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:47:33.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:47:33.703 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:47:09.680+0000 7f306780d640 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:47:33.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:47:33.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:47:33.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:47:33.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:47:33.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:47:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:47:33.811 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:47:33.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:47:33.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:47:33.812 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:47:33.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:33.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:47:33.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:47:33.814 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:33.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:47:33.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:47:33.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:47:33.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:47:33.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:33.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:33.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:47:33.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:47:33.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:47:33.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:47:33.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:47:33.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:47:33.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:47:33.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:47:33.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:47:33.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:47:33.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:47:33.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:33.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:47:33.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:47:33.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:47:33.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:47:33.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:47:33.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:47:33.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:47:33.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:33.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:33.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:47:33.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:47:33.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:47:33.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:47:33.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:47:33.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:33.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:33.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:47:33.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:47:33.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:47:33.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:47:33.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:47:33.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:34.296 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:34.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:34.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:34.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:47:34.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:47:34.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:47:34.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:47:34.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:47:34.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:47:34.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:47:34.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:47:34.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:47:34.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:35.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:47:35.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:47:35.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:47:35.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:47:35.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:47:35.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:47:35.605 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:47:35.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:47:35.617 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:47:35.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:47:35.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:47:35.620 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 29cd7bad-5813-4fe9-a049-c81255fd01a0 2024-09-10T16:47:35.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29cd7bad-5813-4fe9-a049-c81255fd01a0 2024-09-10T16:47:35.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 29cd7bad-5813-4fe9-a049-c81255fd01a0' 2024-09-10T16:47:35.621 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:47:35.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAneOBmL2SlJRAAjfDpZtVMxxMjFvNJ6N8WPQ== 2024-09-10T16:47:35.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAneOBmL2SlJRAAjfDpZtVMxxMjFvNJ6N8WPQ=="}' 2024-09-10T16:47:35.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29cd7bad-5813-4fe9-a049-c81255fd01a0 -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:47:35.904 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:47:35.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:47:35.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQAneOBmL2SlJRAAjfDpZtVMxxMjFvNJ6N8WPQ== --osd-uuid 29cd7bad-5813-4fe9-a049-c81255fd01a0 2024-09-10T16:47:35.942 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:35.940+0000 7f6a7380f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:35.980+0000 7f6a7380f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:35.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:35.981+0000 7f6a7380f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:35.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:35.982+0000 7f6a7380f640 -1 bdev(0x562e404e1c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:47:35.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:35.982+0000 7f6a7380f640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:47:41.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:47:41.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:47:41.386 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:47:41.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:47:41.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:47:41.920 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:47:41.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:47:41.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-10T16:47:41.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:47:41.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:47:41.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:47:41.946 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:41.945+0000 7f708d9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:41.948 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:41.947+0000 7f708d9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:41.949 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:41.948+0000 7f708d9a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:47:42.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:47:42.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:47:42.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:47:42.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:47:42.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:47:42.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:42.351 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:47:42.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:47:42.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:42.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:42.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:43.284 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:43.283+0000 7f708d9a8640 -1 Falling back to public interface 2024-09-10T16:47:43.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:43.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:43.734 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:47:43.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:47:43.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:43.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:44.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:45.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:45.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:45.154 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:47:45.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:47:45.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:45.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:45.355 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:47:45.353+0000 7f708d9a8640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:47:45.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:46.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:46.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:46.614 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:47:46.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:47:46.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:46.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:47.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:48.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:48.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:48.132 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:47:48.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:47:48.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:48.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:48.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:49.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:49.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:49.524 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:47:49.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:47:49.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:49.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:49.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:50.921 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:47:50.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:47:50.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:50.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:51.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:52.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:52.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:52.425 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:47:52.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:47:52.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:52.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:52.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:53.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:53.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:53.816 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:47:53.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:47:53.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:53.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:54.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:55.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:55.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:55.230 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:47:55.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:47:55.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:55.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:55.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:56.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:56.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:56.633 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:47:56.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:47:56.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:56.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:57.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:58.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:58.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:58.150 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:47:58.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:47:58.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:58.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:47:58.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:47:59.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:47:59.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:47:59.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:47:59.990 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:47:59.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:47:59.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:01.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:02.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:02.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:02.009 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:48:02.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:48:02.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:02.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:03.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:04.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:04.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:04.089 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:48:04.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:48:04.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:04.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:05.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:06.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:06.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:06.174 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:48:06.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:48:06.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:06.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:07.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:08.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:08.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:08.013 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:48:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:48:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:08.405 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1240307545,v1:127.0.0.1:6803/1240307545] [v2:127.0.0.1:6804/1240307545,v1:127.0.0.1:6805/1240307545] exists,up 29cd7bad-5813-4fe9-a049-c81255fd01a0 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:48:08.406 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:48:08.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:48:08.485 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:48:08.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:48:08.495 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:47:34.324+0000 7fe0cf437c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:48:08.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:48:08.496 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:47:43.284+0000 7f708d9a8640 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:48:08.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:48:08.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:48:08.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:48:08.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:48:08.603 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:48:08.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:48:08.604 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:48:08.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:48:08.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:48:08.605 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:48:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:48:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:48:08.606 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:08.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:48:08.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:48:08.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:48:08.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:48:08.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:08.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:08.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:48:08.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:48:08.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:48:08.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:48:08.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:48:08.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:48:08.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:48:08.618 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:48:08.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:48:08.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:48:08.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:48:08.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:08.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:48:08.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:48:08.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:08.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:48:08.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:48:08.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:48:08.623 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:48:08.623 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:08.623 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:08.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:48:08.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:48:08.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:48:08.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2024-09-10T16:48:08.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:48:08.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:08.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:08.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27184 2024-09-10T16:48:08.627 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:48:08.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T16:48:08.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:48:08.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-10T16:48:08.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-10T16:48:08.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:09.060 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:09.061 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:48:09.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:09.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:48:09.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:48:09.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:48:09.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:48:09.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:48:09.247 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:48:09.247 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get fsid 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:48:09.328 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:48:09.329 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27184/ceph-mon.a.asok config get mon_host 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-10T16:48:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:48:10.259 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:48:10.260 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:10.260 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:10.260 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:10.260 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:10.260 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:10.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:48:10.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:48:10.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:48:10.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-mon.a.asok 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:48:10.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-mon.a.asok log flush 2024-09-10T16:48:10.361 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:48:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:10.373 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:48:10.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:10.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-10T16:48:10.375 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:48:10.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=48450b13-b424-44bd-b72d-386ffe0fcc44 2024-09-10T16:48:10.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 48450b13-b424-44bd-b72d-386ffe0fcc44' 2024-09-10T16:48:10.375 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 48450b13-b424-44bd-b72d-386ffe0fcc44 2024-09-10T16:48:10.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:48:10.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKeOBmzpwiFxAA6Vbf5Sb56VbgspEJOMdpmw== 2024-09-10T16:48:10.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKeOBmzpwiFxAA6Vbf5Sb56VbgspEJOMdpmw=="}' 2024-09-10T16:48:10.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 48450b13-b424-44bd-b72d-386ffe0fcc44 -i td/test-erasure-code-plugins/0/new.json 2024-09-10T16:48:10.635 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:48:10.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-10T16:48:10.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQBKeOBmzpwiFxAA6Vbf5Sb56VbgspEJOMdpmw== --osd-uuid 48450b13-b424-44bd-b72d-386ffe0fcc44 2024-09-10T16:48:10.674 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:10.672+0000 7fa4cf789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:10.704 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:10.702+0000 7fa4cf789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:10.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:10.704+0000 7fa4cf789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:10.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:10.704+0000 7fa4cf789640 -1 bdev(0x5627f1eebc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:48:10.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:10.704+0000 7fa4cf789640 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-10T16:48:16.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-10T16:48:16.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:48:16.082 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:48:16.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:48:16.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:48:16.624 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:48:16.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:48:16.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=813be837-9095-4b62-bb26-b4f6405f17d6 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.27184/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-10T16:48:16.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:48:16.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:48:16.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:48:16.650 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:16.648+0000 7f18b7e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:16.651 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:16.650+0000 7f18b7e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:16.652 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:16.651+0000 7f18b7e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:48:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:48:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:48:17.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:48:17.036 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:48:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:48:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:48:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:17.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:17.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:18.257 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:18.255+0000 7f18b7e0b640 -1 Falling back to public interface 2024-09-10T16:48:18.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:18.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:18.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:48:18.455 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:48:18.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:18.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:18.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:19.841 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:19.840+0000 7f18b7e0b640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:48:19.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:19.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:19.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:48:19.881 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:48:19.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:19.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:20.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:21.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:21.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:21.445 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:48:21.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:48:21.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:21.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:21.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:22.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:22.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:22.927 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:48:22.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:48:22.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:22.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:23.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:24.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:24.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:24.320 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:48:24.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:48:24.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:24.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:24.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:25.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:25.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:25.744 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:48:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:48:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:26.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:27.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:27.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:27.137 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:48:27.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:48:27.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:27.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:27.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:28.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:28.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:28.565 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:48:28.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:48:28.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:28.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:28.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:29.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:29.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:29.986 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:48:29.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:48:29.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:29.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:30.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:31.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:31.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:31.403 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:48:31.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:48:31.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:31.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:32.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:33.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:33.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:33.245 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:48:33.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:48:33.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:33.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:33.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:34.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:34.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:34.771 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:48:34.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:48:34.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:34.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:35.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:36.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:36.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:36.461 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:48:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:48:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:37.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:37.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:37.901 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:48:37.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:48:37.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:37.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:38.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:39.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:39.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:39.657 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:48:39.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:48:39.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:39.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:40.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:41.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:41.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:41.401 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:48:41.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:48:41.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:41.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:41.794 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/502538413,v1:127.0.0.1:6803/502538413] [v2:127.0.0.1:6804/502538413,v1:127.0.0.1:6805/502538413] exists,up 48450b13-b424-44bd-b72d-386ffe0fcc44 2024-09-10T16:48:41.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:48:41.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:48:41.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:48:41.795 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27184/ceph-osd.0.asok 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-10T16:48:41.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.27184/ceph-osd.0.asok log flush 2024-09-10T16:48:41.875 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:48:41.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-10T16:48:41.885 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:48:09.088+0000 7fe5d9a0fc80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:48:41.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-10T16:48:41.886 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:48:18.256+0000 7f18b7e0b640 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-10T16:48:41.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-10T16:48:41.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:41.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:48:41.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:48:41.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:48:41.993 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:48:41.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:48:41.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:48:41.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:48:41.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:48:41.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:48:41.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:41.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:48:41.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:48:41.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:41.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:48:41.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:48:41.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:48:42.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:48:42.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:42.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:48:42.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code-plugins 0 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:48:42.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:48:42.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:48:42.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:48:42.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:48:42.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:48:42.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:48:42.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:48:42.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:48:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:48:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:48:42.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:42.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:48:42.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T16:48:42.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2024-09-10T16:48:42.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:48:42.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27184 2024-09-10T16:48:42.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27184 2024-09-10T16:48:42.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:48:42.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:48:42.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T16:48:42.015 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T16:48:42.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T16:48:42.083 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-10T16:48:42.083 DEBUG:teuthology.orchestra.run.smithi086:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-10T16:48:42.145 INFO:tasks.workunit.client.0.smithi086.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T16:48:42.151 INFO:tasks.workunit.client.0.smithi086.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-code 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-10T16:48:42.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none ' 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:48:42.153 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:48:42.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:48:42.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:48:42.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:48:42.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:48:42.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:48:42.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:48:42.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:48:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:48:42.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:48:42.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:48:42.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:48:42.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-10T16:48:42.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:48:42.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:42.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55334 2024-09-10T16:48:42.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:48:42.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:48:42.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code 2024-09-10T16:48:42.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:48:42.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:42.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55334 2024-09-10T16:48:42.162 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:48:42.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T16:48:42.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T16:48:42.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:48:42.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code/a 2024-09-10T16:48:42.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:48:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T16:48:42.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:48:42.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:48:42.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:48:42.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55334/ceph-mon.a.asok 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:48:42.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55334/ceph-mon.a.asok config get fsid 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:42.906 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:42.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55334/ceph-mon.a.asok 2024-09-10T16:48:42.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:48:42.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55334/ceph-mon.a.asok config get mon_host 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code/x 2024-09-10T16:48:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:48:43.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:48:43.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:43.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:43.826 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:43.827 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:43.827 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:43.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:48:43.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:48:43.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55334/ceph-mon.a.asok 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-10T16:48:43.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.55334/ceph-mon.a.asok log flush 2024-09-10T16:48:43.923 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:48:43.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-10T16:48:43.932 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:48:42.666+0000 7f4b68bc3c80 0 load: jerasure load: lrc 2024-09-10T16:48:43.933 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-10T16:48:43.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/0 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:48:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:48:43.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:48:43.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/0 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:48:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0500839e-1a47-473a-a6a3-782373b54466 2024-09-10T16:48:43.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0500839e-1a47-473a-a6a3-782373b54466' 2024-09-10T16:48:43.941 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 0500839e-1a47-473a-a6a3-782373b54466 2024-09-10T16:48:43.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:48:43.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBreOBmgIDROBAAVsx0oskvRx9pkWncE6kNnw== 2024-09-10T16:48:43.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBreOBmgIDROBAAVsx0oskvRx9pkWncE6kNnw=="}' 2024-09-10T16:48:43.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0500839e-1a47-473a-a6a3-782373b54466 -i td/test-erasure-code/0/new.json 2024-09-10T16:48:44.209 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:48:44.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/0/new.json 2024-09-10T16:48:44.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBreOBmgIDROBAAVsx0oskvRx9pkWncE6kNnw== --osd-uuid 0500839e-1a47-473a-a6a3-782373b54466 2024-09-10T16:48:44.249 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:44.247+0000 7f83e17a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:44.269 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:44.268+0000 7f83e17a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:44.270 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:44.269+0000 7f83e17a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:44.271 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:44.270+0000 7f83e17a8640 -1 bdev(0x55686364dc00 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:48:44.271 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:44.270+0000 7f83e17a8640 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-10T16:48:49.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-10T16:48:49.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:48:49.690 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:48:49.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:48:49.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:48:50.223 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:48:50.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:48:50.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:48:50.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:48:50.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:48:50.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:48:50.250 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:50.248+0000 7fa4cad1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:50.252 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:50.250+0000 7fa4cad1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:50.253 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:50.252+0000 7fa4cad1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:50.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:48:50.647 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:48:50.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:50.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:51.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:51.606 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:51.605+0000 7fa4cad1e640 -1 Falling back to public interface 2024-09-10T16:48:52.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:52.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:52.056 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:48:52.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:48:52.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:52.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:52.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:53.444 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:48:53.442+0000 7fa4cad1e640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:48:53.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:53.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:53.483 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:48:53.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:48:53.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:53.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:54.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:55.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:55.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:55.065 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:48:55.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:48:55.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:55.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:55.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:56.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:56.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:56.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:48:56.532 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:48:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:56.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:56.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:57.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:57.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:57.951 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:48:57.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:48:57.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:57.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:48:59.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:48:59.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:48:59.345 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:48:59.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:48:59.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:48:59.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:48:59.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:00.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:00.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:00.798 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:49:00.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:49:00.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:00.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:01.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:02.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:02.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:02.223 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:49:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:49:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:02.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:03.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:03.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:03.628 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:49:03.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:49:03.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:03.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:04.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:05.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:05.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:05.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:49:05.026 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:49:05.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:05.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:06.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:06.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:06.897 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:49:06.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:49:06.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:06.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:07.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:08.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:08.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:08.491 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:49:08.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:49:08.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:08.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:09.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:10.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:10.199 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:49:10.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:10.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:49:10.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:10.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:10.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:11.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:11.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:11.895 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:49:11.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:49:11.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:11.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:12.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:13.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:13.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:13.354 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:49:13.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:49:13.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:13.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:14.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:15.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:15.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:15.041 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:49:15.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:49:15.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:15.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:49:15.452 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2843284857,v1:127.0.0.1:6803/2843284857] [v2:127.0.0.1:6804/2843284857,v1:127.0.0.1:6805/2843284857] exists,up 0500839e-1a47-473a-a6a3-782373b54466 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/1 2024-09-10T16:49:15.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:49:15.459 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:49:15.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/1 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2201d32a-f6aa-438b-8ca1-13108baaea91 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2201d32a-f6aa-438b-8ca1-13108baaea91' 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 2201d32a-f6aa-438b-8ca1-13108baaea91 2024-09-10T16:49:15.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:49:15.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLeOBmZq8pHBAAlDTMTkoyTz/0GuVUmZrJKA== 2024-09-10T16:49:15.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLeOBmZq8pHBAAlDTMTkoyTz/0GuVUmZrJKA=="}' 2024-09-10T16:49:15.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2201d32a-f6aa-438b-8ca1-13108baaea91 -i td/test-erasure-code/1/new.json 2024-09-10T16:49:15.953 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:49:15.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/1/new.json 2024-09-10T16:49:15.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLeOBmZq8pHBAAlDTMTkoyTz/0GuVUmZrJKA== --osd-uuid 2201d32a-f6aa-438b-8ca1-13108baaea91 2024-09-10T16:49:15.994 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:15.992+0000 7f80bd82a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:15.995 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:15.994+0000 7f80bd82a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:15.996 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:15.995+0000 7f80bd82a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:15.997 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:15.995+0000 7f80bd82a640 -1 bdev(0x564bf7449c00 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2024-09-10T16:49:15.997 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:15.996+0000 7f80bd82a640 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-10T16:49:21.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-10T16:49:21.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:49:21.639 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T16:49:21.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T16:49:21.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:49:22.188 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T16:49:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T16:49:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:49:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:49:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:49:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:49:22.215 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:22.214+0000 7fc0dba1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:22.235 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:22.233+0000 7fc0dba1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:22.236 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:22.234+0000 7fc0dba1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:22.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T16:49:22.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:49:22.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T16:49:22.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:49:22.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:49:22.649 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:49:22.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:22.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:49:22.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:22.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:23.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:23.575 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:23.574+0000 7fc0dba1a640 -1 Falling back to public interface 2024-09-10T16:49:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:24.067 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:49:24.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:49:24.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:24.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:24.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:25.388 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:25.386+0000 7fc0dba1a640 -1 osd.1 0 log_to_monitors true 2024-09-10T16:49:25.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:25.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:25.485 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:49:25.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:49:25.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:25.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:26.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:27.118 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:49:27.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:49:27.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:27.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:27.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:28.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:28.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:28.571 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:49:28.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:49:28.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:28.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:28.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:29.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:29.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:29.994 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:49:29.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:49:29.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:29.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:30.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:31.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:31.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:31.403 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:49:31.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:49:31.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:31.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:31.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:32.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:32.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:32.829 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:49:32.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:49:32.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:32.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:33.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:34.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:34.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:34.289 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:49:34.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:49:34.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:34.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:34.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:35.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:35.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:35.711 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:49:35.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:49:35.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:35.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:36.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:37.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:37.227 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:49:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:49:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:37.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:38.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:38.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:38.621 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:49:38.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:49:38.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:38.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:39.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:40.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:40.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:40.382 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:49:40.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:49:40.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:40.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:41.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:42.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:42.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:42.232 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:49:42.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:49:42.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:42.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:43.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:44.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:44.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:44.307 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:49:44.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:49:44.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:44.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:45.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:46.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:46.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:46.134 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:49:46.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:49:46.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:46.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:46.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:47.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:47.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:47.988 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:49:47.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:49:47.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:47.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:48.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:49.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:49.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:49.479 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:49:49.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:49:49.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:49.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:49:49.907 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1893643735,v1:127.0.0.1:6811/1893643735] [v2:127.0.0.1:6812/1893643735,v1:127.0.0.1:6813/1893643735] exists,up 2201d32a-f6aa-438b-8ca1-13108baaea91 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:49:49.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/2 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:49:49.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:49:49.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:49:49.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:49:49.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:49:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/2 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3593a7c2-98a6-4146-9a57-d2d4c8b508e5 2024-09-10T16:49:49.915 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 3593a7c2-98a6-4146-9a57-d2d4c8b508e5 2024-09-10T16:49:49.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3593a7c2-98a6-4146-9a57-d2d4c8b508e5' 2024-09-10T16:49:49.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:49:49.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCteOBmCFpFNxAAZw4inf53UpRlL1QUu3oSnA== 2024-09-10T16:49:49.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCteOBmCFpFNxAAZw4inf53UpRlL1QUu3oSnA=="}' 2024-09-10T16:49:49.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3593a7c2-98a6-4146-9a57-d2d4c8b508e5 -i td/test-erasure-code/2/new.json 2024-09-10T16:49:50.389 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:49:50.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/2/new.json 2024-09-10T16:49:50.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCteOBmCFpFNxAAZw4inf53UpRlL1QUu3oSnA== --osd-uuid 3593a7c2-98a6-4146-9a57-d2d4c8b508e5 2024-09-10T16:49:50.429 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:50.427+0000 7f02d23c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:50.433 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:50.432+0000 7f02d23c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:50.434 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:50.433+0000 7f02d23c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:50.434 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:50.433+0000 7f02d23c3640 -1 bdev(0x558839b73c00 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2024-09-10T16:49:50.434 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:50.433+0000 7f02d23c3640 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-10T16:49:56.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-10T16:49:56.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:49:56.102 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T16:49:56.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T16:49:56.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:49:56.635 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T16:49:56.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T16:49:56.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:49:56.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:49:56.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:49:56.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:49:56.661 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:56.660+0000 7fdce154b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:56.685 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:56.684+0000 7fdce154b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:56.687 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:56.685+0000 7fdce154b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:57.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:49:57.084 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:49:57.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:57.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:49:57.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:57.782 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:57.781+0000 7fdce154b640 -1 Falling back to public interface 2024-09-10T16:49:58.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:58.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:58.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:49:58.502 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:49:58.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:58.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:49:58.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:49:59.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:49:59.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:49:59.889 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:49:59.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:49:59.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:49:59.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:49:59.904 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:49:59.903+0000 7fdce154b640 -1 osd.2 0 log_to_monitors true 2024-09-10T16:50:00.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:01.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:01.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:01.424 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:50:01.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:50:01.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:01.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:01.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:02.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:02.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:02.859 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:50:02.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:50:02.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:02.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:03.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:04.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:04.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:04.252 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:50:04.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:50:04.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:04.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:04.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:05.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:05.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:05.743 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:50:05.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:50:05.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:05.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:06.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:07.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:07.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:07.240 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:50:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:50:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:07.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:08.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:08.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:08.730 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:50:08.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:50:08.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:08.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:09.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:10.126 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:50:10.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:50:10.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:10.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:10.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:11.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:11.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:11.527 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:50:11.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:50:11.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:11.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:12.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:13.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:13.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:13.385 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:50:13.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:50:13.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:13.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:14.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:15.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:15.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:15.434 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:50:15.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:50:15.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:15.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:16.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:17.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:17.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:17.220 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:50:17.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:50:17.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:17.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:18.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:19.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:19.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:19.031 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:50:19.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:50:19.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:19.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:19.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:20.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:20.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:20.784 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:50:20.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:50:20.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:20.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:21.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:22.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:22.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:22.956 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:50:22.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:50:22.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:22.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:50:23.373 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2069614660,v1:127.0.0.1:6819/2069614660] [v2:127.0.0.1:6820/2069614660,v1:127.0.0.1:6821/2069614660] exists,up 3593a7c2-98a6-4146-9a57-d2d4c8b508e5 2024-09-10T16:50:23.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/3 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:50:23.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:50:23.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:50:23.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/3 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=24dc7c4c-ac60-4a9b-a786-0215296fb72d 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 24dc7c4c-ac60-4a9b-a786-0215296fb72d' 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 24dc7c4c-ac60-4a9b-a786-0215296fb72d 2024-09-10T16:50:23.380 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:50:23.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPeOBmWyp4FxAAVF0xDTCsrQ2mIsH4ctjRAQ== 2024-09-10T16:50:23.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPeOBmWyp4FxAAVF0xDTCsrQ2mIsH4ctjRAQ=="}' 2024-09-10T16:50:23.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 24dc7c4c-ac60-4a9b-a786-0215296fb72d -i td/test-erasure-code/3/new.json 2024-09-10T16:50:23.949 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:50:23.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/3/new.json 2024-09-10T16:50:23.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPeOBmWyp4FxAAVF0xDTCsrQ2mIsH4ctjRAQ== --osd-uuid 24dc7c4c-ac60-4a9b-a786-0215296fb72d 2024-09-10T16:50:23.989 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:23.988+0000 7f5960ba6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:24.042 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:24.040+0000 7f5960ba6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:24.043 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:24.041+0000 7f5960ba6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:24.043 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:24.042+0000 7f5960ba6640 -1 bdev(0x5640d37d1c00 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2024-09-10T16:50:24.043 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:24.042+0000 7f5960ba6640 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-10T16:50:29.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-10T16:50:29.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:50:29.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T16:50:29.762 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T16:50:29.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:50:30.296 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T16:50:30.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T16:50:30.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:50:30.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:50:30.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:50:30.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:50:30.323 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:30.322+0000 7fcb65732640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:30.357 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:30.356+0000 7fcb65732640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:30.358 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:30.357+0000 7fcb65732640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:30.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:50:30.724 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:50:30.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:30.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:31.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:31.455 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:31.454+0000 7fcb65732640 -1 Falling back to public interface 2024-09-10T16:50:32.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:32.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:32.103 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:50:32.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:50:32.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:32.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:32.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:33.123 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:33.121+0000 7fcb65732640 -1 osd.3 0 log_to_monitors true 2024-09-10T16:50:33.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:33.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:33.538 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:50:33.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:50:33.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:33.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:33.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:34.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:34.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:34.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:50:34.969 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:50:34.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:34.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:35.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:36.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:36.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:36.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:50:36.374 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:50:36.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:36.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:36.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:37.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:37.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:37.794 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:50:37.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:50:37.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:37.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:38.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:39.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:39.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:39.400 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:50:39.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:50:39.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:39.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:39.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:40.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:40.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:40.793 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:50:40.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:50:40.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:40.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:41.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:42.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:42.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:42.268 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:50:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:50:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:42.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:43.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:43.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:43.659 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:50:43.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:50:43.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:43.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:44.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:45.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:45.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:45.172 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:50:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:50:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:45.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:46.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:46.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:46.636 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:50:46.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:50:46.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:46.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:47.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:48.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:48.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:48.784 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:50:48.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:50:48.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:48.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:49.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:50.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:50.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:50.217 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:50:50.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:50:50.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:50.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:50.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:51.949 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:50:51.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:50:51.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:51.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:52.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:53.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:53.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:53.546 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:50:53.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:50:53.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:53.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:54.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:55.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:55.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:55.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:50:55.200 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:50:55.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:55.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:55.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:50:56.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:50:56.828 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:50:56.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:50:56.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:50:56.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:50:56.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:50:57.281 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3584868081,v1:127.0.0.1:6827/3584868081] [v2:127.0.0.1:6828/3584868081,v1:127.0.0.1:6829/3584868081] exists,up 24dc7c4c-ac60-4a9b-a786-0215296fb72d 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:50:57.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/4 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:50:57.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:50:57.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/4 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab345450-da01-4306-a191-8a235ab59736 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 ab345450-da01-4306-a191-8a235ab59736' 2024-09-10T16:50:57.288 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 ab345450-da01-4306-a191-8a235ab59736 2024-09-10T16:50:57.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:50:57.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxeOBmqW3+ERAAqwUVjqaZ1emdTKtDjaU45Q== 2024-09-10T16:50:57.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxeOBmqW3+ERAAqwUVjqaZ1emdTKtDjaU45Q=="}' 2024-09-10T16:50:57.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab345450-da01-4306-a191-8a235ab59736 -i td/test-erasure-code/4/new.json 2024-09-10T16:50:57.789 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:50:57.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/4/new.json 2024-09-10T16:50:57.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxeOBmqW3+ERAAqwUVjqaZ1emdTKtDjaU45Q== --osd-uuid ab345450-da01-4306-a191-8a235ab59736 2024-09-10T16:50:57.829 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:57.828+0000 7f3da4702640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:57.907 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:57.906+0000 7f3da4702640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:57.908 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:57.907+0000 7f3da4702640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:50:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:57.907+0000 7f3da4702640 -1 bdev(0x555beb539c00 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2024-09-10T16:50:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:50:57.907+0000 7f3da4702640 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-10T16:51:03.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-10T16:51:03.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:51:03.899 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T16:51:03.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T16:51:03.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:51:04.462 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T16:51:04.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T16:51:04.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:51:04.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:51:04.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:51:04.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:51:04.488 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:04.487+0000 7f33969cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:04.521 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:04.519+0000 7f33969cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:04.521 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:04.520+0000 7f33969cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:04.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:51:04.877 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:51:04.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:04.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:05.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:05.646 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:05.645+0000 7f33969cc640 -1 Falling back to public interface 2024-09-10T16:51:06.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:06.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:06.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:51:06.294 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:51:06.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:06.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:06.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:07.323 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:07.322+0000 7f33969cc640 -1 osd.4 0 log_to_monitors true 2024-09-10T16:51:07.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:07.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:07.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:51:07.722 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:51:07.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:07.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:08.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:09.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:09.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:09.220 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:51:09.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:51:09.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:09.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:09.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:10.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:10.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:10.609 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:51:10.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:51:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:11.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:12.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:12.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:12.133 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:51:12.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:51:12.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:12.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:12.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:13.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:13.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:13.562 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:51:13.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:51:13.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:13.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:14.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:15.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:15.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:15.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:51:15.025 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:51:15.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:15.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:15.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:16.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:16.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:16.456 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:51:16.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:51:16.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:16.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:16.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:17.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:17.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:17.854 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:51:17.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:51:17.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:17.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:18.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:19.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:19.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:19.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:51:19.369 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:51:19.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:19.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:20.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:21.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:21.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:21.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:51:21.008 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:51:21.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:21.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:21.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:22.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:22.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:22.710 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:51:22.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:51:22.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:22.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:23.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:24.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:24.454 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:51:24.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:51:24.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:24.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:25.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:26.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:26.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:26.282 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:51:26.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:51:26.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:26.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:27.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:28.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:28.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:28.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:51:28.363 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:51:28.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:28.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:29.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:30.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:30.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:30.422 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:51:30.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:51:30.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:30.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T16:51:30.837 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/14866291,v1:127.0.0.1:6835/14866291] [v2:127.0.0.1:6836/14866291,v1:127.0.0.1:6837/14866291] exists,up ab345450-da01-4306-a191-8a235ab59736 2024-09-10T16:51:30.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:51:30.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:51:30.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/5 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-10T16:51:30.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:51:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/5 2024-09-10T16:51:30.844 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:51:30.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=50cee0f8-81be-4ca4-afd9-8355fe76a6ff 2024-09-10T16:51:30.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 50cee0f8-81be-4ca4-afd9-8355fe76a6ff' 2024-09-10T16:51:30.845 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 50cee0f8-81be-4ca4-afd9-8355fe76a6ff 2024-09-10T16:51:30.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:51:30.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASeeBmpAohMxAAKJhizn/bROISXMUW2+T63A== 2024-09-10T16:51:30.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASeeBmpAohMxAAKJhizn/bROISXMUW2+T63A=="}' 2024-09-10T16:51:30.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 50cee0f8-81be-4ca4-afd9-8355fe76a6ff -i td/test-erasure-code/5/new.json 2024-09-10T16:51:31.426 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:51:31.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/5/new.json 2024-09-10T16:51:31.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASeeBmpAohMxAAKJhizn/bROISXMUW2+T63A== --osd-uuid 50cee0f8-81be-4ca4-afd9-8355fe76a6ff 2024-09-10T16:51:31.467 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:31.466+0000 7f08f69cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:31.469 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:31.468+0000 7f08f69cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:31.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:31.469+0000 7f08f69cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:31.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:31.469+0000 7f08f69cc640 -1 bdev(0x55ddcc273c00 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2024-09-10T16:51:31.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:31.469+0000 7f08f69cc640 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-10T16:51:37.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-10T16:51:37.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:51:37.340 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T16:51:37.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T16:51:37.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:51:37.871 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T16:51:37.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T16:51:37.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:51:37.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:51:37.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:51:37.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:51:37.898 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:37.897+0000 7f60e440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:37.916 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:37.914+0000 7f60e440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:37.918 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:37.915+0000 7f60e440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:38.330 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:51:38.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:51:38.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:38.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:38.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:39.513 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:39.512+0000 7f60e440d640 -1 Falling back to public interface 2024-09-10T16:51:39.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:39.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:39.765 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:51:39.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:51:39.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:39.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:40.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:41.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:41.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:41.183 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:51:41.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:51:41.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:41.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:41.381 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:51:41.379+0000 7f60e440d640 -1 osd.5 0 log_to_monitors true 2024-09-10T16:51:41.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:42.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:42.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:42.591 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:51:42.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:51:42.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:42.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:43.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:44.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:44.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:44.030 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:51:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:51:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:44.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:51:45.430 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:51:45.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:45.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:45.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:46.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:46.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:46.855 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:51:46.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:51:46.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:46.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:47.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:48.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:48.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:48.344 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:51:48.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:51:48.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:48.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:48.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:49.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:49.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:49.750 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:51:49.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:51:49.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:49.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:50.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:51.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:51.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:51.297 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:51:51.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:51:51.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:51.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:51.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:52.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:52.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:52.689 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:51:52.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:51:52.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:52.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:53.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:54.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:54.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:54.204 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:51:54.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:51:54.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:54.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:55.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:56.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:56.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:56.200 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:51:56.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:51:56.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:56.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:57.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:51:58.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:51:58.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:51:58.530 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:51:58.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:51:58.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:51:58.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:51:59.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:00.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:00.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:00.283 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:52:00.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:52:00.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:00.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:52:01.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:02.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:02.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:02.062 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:52:02.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:52:02.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:02.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:52:02.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:03.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:03.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:03.841 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:52:03.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:52:03.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:03.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:52:04.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:05.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:05.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:05.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:52:05.467 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:52:05.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:05.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:52:05.859 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2246480983,v1:127.0.0.1:6843/2246480983] [v2:127.0.0.1:6844/2246480983,v1:127.0.0.1:6845/2246480983] exists,up 50cee0f8-81be-4ca4-afd9-8355fe76a6ff 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T16:52:05.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/6 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:52:05.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:52:05.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/6 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 36b78b87-c9dd-4080-8317-c51c3be1fc24 2024-09-10T16:52:05.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36b78b87-c9dd-4080-8317-c51c3be1fc24 2024-09-10T16:52:05.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 36b78b87-c9dd-4080-8317-c51c3be1fc24' 2024-09-10T16:52:05.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:52:05.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1eeBmddprNBAA1PsexxC1Ezj5gwZaX+m9hg== 2024-09-10T16:52:05.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1eeBmddprNBAA1PsexxC1Ezj5gwZaX+m9hg=="}' 2024-09-10T16:52:05.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36b78b87-c9dd-4080-8317-c51c3be1fc24 -i td/test-erasure-code/6/new.json 2024-09-10T16:52:06.430 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:52:06.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/6/new.json 2024-09-10T16:52:06.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1eeBmddprNBAA1PsexxC1Ezj5gwZaX+m9hg== --osd-uuid 36b78b87-c9dd-4080-8317-c51c3be1fc24 2024-09-10T16:52:06.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:06.468+0000 7f0f04e5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:06.508 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:06.506+0000 7f0f04e5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:06.509 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:06.507+0000 7f0f04e5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:06.509 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:06.508+0000 7f0f04e5e640 -1 bdev(0x55d5d0311c00 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2024-09-10T16:52:06.509 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:06.508+0000 7f0f04e5e640 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-10T16:52:13.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-10T16:52:13.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:52:13.068 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T16:52:13.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T16:52:13.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:52:13.602 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T16:52:13.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T16:52:13.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:52:13.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:52:13.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:52:13.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:52:13.629 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:13.628+0000 7fcb91e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:13.663 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:13.662+0000 7fcb91e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:13.664 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:13.663+0000 7fcb91e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T16:52:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:52:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T16:52:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:52:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:52:14.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:14.045 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:52:14.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:52:14.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:14.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:14.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:15.000 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:14.999+0000 7fcb91e13640 -1 Falling back to public interface 2024-09-10T16:52:15.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:15.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:15.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:52:15.505 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:52:15.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:15.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:15.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:16.851 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:16.849+0000 7fcb91e13640 -1 osd.6 0 log_to_monitors true 2024-09-10T16:52:16.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:16.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:16.902 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:52:16.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:52:16.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:16.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:17.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:18.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:18.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:18.557 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:52:18.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:52:18.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:18.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:18.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:19.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:19.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:19.950 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:52:19.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:52:19.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:19.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:20.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:21.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:21.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:21.468 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:52:21.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:52:21.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:21.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:22.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:23.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:23.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:23.039 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:52:23.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:52:23.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:23.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:23.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:24.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:24.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:24.432 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:52:24.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:52:24.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:24.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:24.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:25.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:25.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:25.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:52:25.821 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:52:25.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:25.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:26.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:27.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:27.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:27.362 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:52:27.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:52:27.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:27.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:27.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:28.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:28.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:28.822 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:52:28.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:52:28.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:28.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:29.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:30.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:30.359 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:52:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:52:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:30.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:31.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:32.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:32.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:32.213 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:52:32.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:52:32.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:32.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:32.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:33.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:33.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:33.980 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:52:33.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:52:33.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:33.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:34.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:35.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:35.448 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:52:35.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:35.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:52:35.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:35.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:36.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:37.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:37.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:37.264 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:52:37.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:52:37.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:37.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:38.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:39.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:39.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:39.016 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:52:39.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:52:39.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:39.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:39.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:40.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:40.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:40.930 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:52:40.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:52:40.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:40.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:52:41.311 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4249262285,v1:127.0.0.1:6851/4249262285] [v2:127.0.0.1:6852/4249262285,v1:127.0.0.1:6853/4249262285] exists,up 36b78b87-c9dd-4080-8317-c51c3be1fc24 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:52:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/7 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:52:41.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:52:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/7 2024-09-10T16:52:41.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:52:41.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8c8bac15-adec-40fd-8669-591aa6e6ca98 2024-09-10T16:52:41.319 INFO:tasks.workunit.client.0.smithi086.stdout:add osd7 8c8bac15-adec-40fd-8669-591aa6e6ca98 2024-09-10T16:52:41.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 8c8bac15-adec-40fd-8669-591aa6e6ca98' 2024-09-10T16:52:41.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:52:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZeeBmL83aExAAr0ZhNXP4ZoctZVSf34xX3Q== 2024-09-10T16:52:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZeeBmL83aExAAr0ZhNXP4ZoctZVSf34xX3Q=="}' 2024-09-10T16:52:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8c8bac15-adec-40fd-8669-591aa6e6ca98 -i td/test-erasure-code/7/new.json 2024-09-10T16:52:41.819 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:52:41.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/7/new.json 2024-09-10T16:52:41.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZeeBmL83aExAAr0ZhNXP4ZoctZVSf34xX3Q== --osd-uuid 8c8bac15-adec-40fd-8669-591aa6e6ca98 2024-09-10T16:52:41.860 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:41.858+0000 7f45dc813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:41.919 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:41.917+0000 7f45dc813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:41.919 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:41.918+0000 7f45dc813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:41.920 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:41.918+0000 7f45dc813640 -1 bdev(0x5642f53e9c00 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2024-09-10T16:52:41.920 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:41.919+0000 7f45dc813640 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-10T16:52:47.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-10T16:52:47.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:52:47.898 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd7 key to auth repository 2024-09-10T16:52:47.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-10T16:52:47.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:52:48.471 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.7 2024-09-10T16:52:48.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-10T16:52:48.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:52:48.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:52:48.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:52:48.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:52:48.498 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:48.497+0000 7ff09e064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:48.512 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:48.510+0000 7ff09e064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:48.513 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:48.511+0000 7ff09e064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:52:48.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:48.918 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:52:48.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:52:48.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:48.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:49.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:50.111 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:50.110+0000 7ff09e064640 -1 Falling back to public interface 2024-09-10T16:52:50.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:50.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:50.336 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:52:50.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:52:50.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:50.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:50.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:51.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:51.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:51.764 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:52:51.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:52:51.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:51.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:52.032 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:52:52.030+0000 7ff09e064640 -1 osd.7 0 log_to_monitors true 2024-09-10T16:52:52.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:53.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:53.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:53.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:52:53.318 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:52:53.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:53.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:53.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:54.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:54.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:54.713 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:52:54.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:52:54.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:54.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:55.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:56.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:56.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:56.276 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:52:56.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:52:56.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:56.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:56.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:57.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:57.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:57.800 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:52:57.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:52:57.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:57.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:58.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:52:59.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:52:59.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:52:59.374 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:52:59.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:52:59.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:52:59.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:52:59.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:00.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:00.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:00.762 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:53:00.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:53:00.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:00.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:01.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:02.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:02.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:02.285 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:53:02.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:53:02.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:02.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:02.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:03.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:03.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:03.708 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:53:03.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:53:03.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:03.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:04.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:05.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:05.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:05.236 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:53:05.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:53:05.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:05.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:05.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:06.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:06.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:06.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:53:06.673 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:53:06.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:06.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:07.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:08.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:08.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:08.460 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:53:08.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:53:08.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:08.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:09.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:10.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:10.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:10.356 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:53:10.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:53:10.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:10.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:11.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:12.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:12.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:12.210 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:53:12.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:53:12.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:12.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:12.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:13.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:13.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:13.781 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:53:13.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:53:13.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:13.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:14.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:15.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:15.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:15.494 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:53:15.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:53:15.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:15.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T16:53:16.011 INFO:tasks.workunit.client.0.smithi086.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2325554561,v1:127.0.0.1:6859/2325554561] [v2:127.0.0.1:6860/2325554561,v1:127.0.0.1:6861/2325554561] exists,up 8c8bac15-adec-40fd-8669-591aa6e6ca98 2024-09-10T16:53:16.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-10T16:53:16.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/8 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-10T16:53:16.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:53:16.020 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:53:16.021 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:53:16.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:53:16.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:53:16.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:53:16.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:53:16.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:53:16.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:53:16.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:53:16.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:53:16.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:53:16.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:53:16.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:53:16.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:53:16.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:53:16.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/8 2024-09-10T16:53:16.024 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:53:16.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2 2024-09-10T16:53:16.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2' 2024-09-10T16:53:16.024 INFO:tasks.workunit.client.0.smithi086.stdout:add osd8 7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2 2024-09-10T16:53:16.025 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:53:16.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8eeBmNDXpARAAs2ZCbp6rrYU9R+A/Vbz9Ag== 2024-09-10T16:53:16.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8eeBmNDXpARAAs2ZCbp6rrYU9R+A/Vbz9Ag=="}' 2024-09-10T16:53:16.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2 -i td/test-erasure-code/8/new.json 2024-09-10T16:53:16.480 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:53:16.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/8/new.json 2024-09-10T16:53:16.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8eeBmNDXpARAAs2ZCbp6rrYU9R+A/Vbz9Ag== --osd-uuid 7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2 2024-09-10T16:53:16.522 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:16.520+0000 7fb7b372e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:16.523 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:16.522+0000 7fb7b372e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:16.524 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:16.523+0000 7fb7b372e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:16.525 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:16.523+0000 7fb7b372e640 -1 bdev(0x55c5d3549c00 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2024-09-10T16:53:16.525 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:16.523+0000 7fb7b372e640 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-10T16:53:23.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-10T16:53:23.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:53:23.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-10T16:53:23.299 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd8 key to auth repository 2024-09-10T16:53:23.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:53:23.841 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.8 2024-09-10T16:53:23.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-10T16:53:23.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:53:23.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:53:23.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:53:23.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:53:23.868 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:23.866+0000 7fe5dcbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:23.881 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:23.880+0000 7fe5dcbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:23.882 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:23.881+0000 7fe5dcbae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:24.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-10T16:53:24.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:53:24.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-10T16:53:24.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:24.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:24.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:25.474 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:25.473+0000 7fe5dcbae640 -1 Falling back to public interface 2024-09-10T16:53:25.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:25.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:25.674 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:53:25.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:53:25.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:25.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:26.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:27.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:27.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:27.090 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:53:27.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:53:27.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:27.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:27.219 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:27.218+0000 7fe5dcbae640 -1 osd.8 0 log_to_monitors true 2024-09-10T16:53:27.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:28.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:28.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:28.549 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:53:28.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:53:28.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:28.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:29.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:30.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:30.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:30.088 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:53:30.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:53:30.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:30.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:30.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:31.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:31.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:31.675 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:53:31.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:53:31.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:31.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:32.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:33.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:33.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:33.282 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:53:33.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:53:33.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:33.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:33.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:34.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:34.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:34.692 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:53:34.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:53:34.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:34.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:35.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:36.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:36.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:36.220 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:53:36.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:53:36.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:36.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:36.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:37.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:37.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:37.611 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:53:37.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:53:37.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:37.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:38.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:39.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:39.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:39.118 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:53:39.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:53:39.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:39.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:39.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:40.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:40.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:40.708 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:53:40.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:53:40.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:40.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:41.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:42.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:42.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:42.253 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:53:42.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:53:42.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:42.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:43.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:44.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:44.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:44.394 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:53:44.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:53:44.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:44.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:44.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:45.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:45.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:45.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:53:45.885 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:53:45.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:45.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:46.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:47.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:47.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:47.319 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:53:47.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:53:47.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:47.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:48.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:49.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:49.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:49.553 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:53:49.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:53:49.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:49.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:50.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:53:51.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:53:51.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:53:51.390 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:53:51.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:53:51.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:53:51.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T16:53:51.847 INFO:tasks.workunit.client.0.smithi086.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1439352050,v1:127.0.0.1:6867/1439352050] [v2:127.0.0.1:6868/1439352050,v1:127.0.0.1:6869/1439352050] exists,up 7777cc9e-6bdb-49d0-a6f8-abc6ea435ff2 2024-09-10T16:53:51.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/9 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:53:51.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:53:51.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:53:51.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/9 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a85f522e-e117-40bd-8988-e4dd30d5ffcf 2024-09-10T16:53:51.856 INFO:tasks.workunit.client.0.smithi086.stdout:add osd9 a85f522e-e117-40bd-8988-e4dd30d5ffcf 2024-09-10T16:53:51.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 a85f522e-e117-40bd-8988-e4dd30d5ffcf' 2024-09-10T16:53:51.857 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:53:51.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCfeeBmMqu3MxAA8rVQhf83nj49J5/R7TmRgg== 2024-09-10T16:53:51.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCfeeBmMqu3MxAA8rVQhf83nj49J5/R7TmRgg=="}' 2024-09-10T16:53:51.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a85f522e-e117-40bd-8988-e4dd30d5ffcf -i td/test-erasure-code/9/new.json 2024-09-10T16:53:52.345 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:53:52.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/9/new.json 2024-09-10T16:53:52.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfeeBmMqu3MxAA8rVQhf83nj49J5/R7TmRgg== --osd-uuid a85f522e-e117-40bd-8988-e4dd30d5ffcf 2024-09-10T16:53:52.385 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:52.383+0000 7f2df091b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:52.424 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:52.422+0000 7f2df091b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:52.424 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:52.423+0000 7f2df091b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:52.425 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:52.423+0000 7f2df091b640 -1 bdev(0x55c824391c00 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2024-09-10T16:53:52.425 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:52.424+0000 7f2df091b640 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-10T16:53:59.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-10T16:53:59.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:53:59.254 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd9 key to auth repository 2024-09-10T16:53:59.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-10T16:53:59.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:53:59.784 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.9 2024-09-10T16:53:59.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-10T16:53:59.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:53:59.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:53:59.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:53:59.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:53:59.810 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:59.809+0000 7fa5a860b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:59.830+0000 7fa5a860b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:53:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:53:59.831+0000 7fa5a860b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:00.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-10T16:54:00.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:54:00.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T16:54:00.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:54:00.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:54:00.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:00.274 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:54:00.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:54:00.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:00.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:00.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:01.436 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:01.434+0000 7fa5a860b640 -1 Falling back to public interface 2024-09-10T16:54:01.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:01.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:01.700 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:54:01.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:54:01.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:01.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:02.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:03.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:03.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:03.096 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:54:03.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:54:03.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:03.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:03.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:03.570 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:03.568+0000 7fa5a860b640 -1 osd.9 0 log_to_monitors true 2024-09-10T16:54:04.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:04.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:04.489 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:54:04.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:54:04.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:04.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:05.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:06.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:06.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:06.021 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:54:06.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:54:06.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:06.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:06.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:07.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:07.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:07.434 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:54:07.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:54:07.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:07.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:07.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:08.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:08.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:08.843 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:54:08.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:54:08.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:08.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:09.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:10.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:10.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:10.241 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:54:10.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:54:10.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:10.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:10.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:11.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:11.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:11.753 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:54:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:54:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:12.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:13.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:13.283 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:54:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:54:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:13.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:14.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:14.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:14.691 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:54:14.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:54:14.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:14.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:15.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:16.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:16.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:16.081 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:54:16.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:54:16.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:16.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:16.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:17.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:17.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:17.926 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:54:17.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:54:17.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:17.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:18.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:19.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:19.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:19.471 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:54:19.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:54:19.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:19.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:20.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:21.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:21.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:21.425 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:54:21.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:54:21.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:21.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:21.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:22.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:22.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:22.913 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:54:22.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:54:22.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:22.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:23.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:24.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:24.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:24.716 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:54:24.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:54:24.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:24.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:25.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:26.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:26.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:26.529 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:54:26.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:54:26.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:26.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:27.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:28.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:28.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:28.268 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T16:54:28.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T16:54:28.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:28.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:54:28.707 INFO:tasks.workunit.client.0.smithi086.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2451796053,v1:127.0.0.1:6875/2451796053] [v2:127.0.0.1:6876/2451796053,v1:127.0.0.1:6877/2451796053] exists,up a85f522e-e117-40bd-8988-e4dd30d5ffcf 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2024-09-10T16:54:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/10 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:54:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:54:28.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/10 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:54:28.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=071c61ef-4e21-4e81-8b71-acd727cea393 2024-09-10T16:54:28.715 INFO:tasks.workunit.client.0.smithi086.stdout:add osd10 071c61ef-4e21-4e81-8b71-acd727cea393 2024-09-10T16:54:28.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 071c61ef-4e21-4e81-8b71-acd727cea393' 2024-09-10T16:54:28.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:54:28.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEeeBmmu9jKxAA8c7p6tcKNRzm3IgQ7nbXjQ== 2024-09-10T16:54:28.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEeeBmmu9jKxAA8c7p6tcKNRzm3IgQ7nbXjQ=="}' 2024-09-10T16:54:28.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 071c61ef-4e21-4e81-8b71-acd727cea393 -i td/test-erasure-code/10/new.json 2024-09-10T16:54:29.381 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:54:29.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/10/new.json 2024-09-10T16:54:29.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEeeBmmu9jKxAA8c7p6tcKNRzm3IgQ7nbXjQ== --osd-uuid 071c61ef-4e21-4e81-8b71-acd727cea393 2024-09-10T16:54:29.419 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:29.418+0000 7f51fd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:29.421 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:29.420+0000 7f51fd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:29.422 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:29.421+0000 7f51fd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:29.422 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:29.421+0000 7f51fd20d640 -1 bdev(0x556887641c00 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2024-09-10T16:54:29.423 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:29.421+0000 7f51fd20d640 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-10T16:54:35.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-10T16:54:35.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:54:35.931 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd10 key to auth repository 2024-09-10T16:54:35.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2024-09-10T16:54:35.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:54:36.491 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.10 2024-09-10T16:54:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2024-09-10T16:54:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:54:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:54:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:54:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:54:36.518 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:36.516+0000 7fda09fbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:36.541 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:36.539+0000 7fda09fbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:36.542 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:36.540+0000 7fda09fbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:54:36.938 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:54:36.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:36.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:54:36.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:36.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:37.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:37.901 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:37.900+0000 7fda09fbb640 -1 Falling back to public interface 2024-09-10T16:54:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:54:38.375 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:54:38.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:38.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:38.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:39.739 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:54:39.737+0000 7fda09fbb640 -1 osd.10 0 log_to_monitors true 2024-09-10T16:54:39.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:39.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:39.810 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:54:39.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:54:39.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:39.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:40.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:41.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:41.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:41.444 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:54:41.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:54:41.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:41.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:41.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:42.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:42.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:42.937 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:54:42.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:54:42.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:42.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:43.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:44.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:44.350 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:54:44.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:44.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:54:44.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:44.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:44.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:45.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:45.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:45.786 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:54:45.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:54:45.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:45.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:46.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:47.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:47.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:47.293 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:54:47.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:54:47.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:47.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:47.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:48.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:48.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:48.714 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:54:48.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:54:48.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:48.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:49.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:50.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:50.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:50.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:54:50.110 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:54:50.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:50.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:50.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:51.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:51.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:51.755 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:54:51.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:54:51.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:51.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:52.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:53.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:53.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:53.153 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:54:53.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:54:53.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:53.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:53.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:54.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:54.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:54.915 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:54:54.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:54:54.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:54.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:55.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:56.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:56.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:56.688 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:54:56.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:54:56.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:56.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:57.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:54:58.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:54:58.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:54:58.490 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:54:58.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:54:58.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:54:58.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:54:59.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:00.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:00.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:00.412 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:55:00.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:55:00.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:00.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:55:01.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:02.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:02.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:02.543 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:55:02.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:55:02.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:02.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:55:02.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:03.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:03.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:03.945 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T16:55:03.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T16:55:03.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:03.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2024-09-10T16:55:04.406 INFO:tasks.workunit.client.0.smithi086.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/914301550,v1:127.0.0.1:6883/914301550] [v2:127.0.0.1:6884/914301550,v1:127.0.0.1:6885/914301550] exists,up 071c61ef-4e21-4e81-8b71-acd727cea393 2024-09-10T16:55:04.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:55:04.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:55:04.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:55:04.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-10T16:55:04.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-10T16:55:04.811 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' does not exist 2024-09-10T16:55:04.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-10T16:55:04.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T16:55:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T16:55:05.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:55:06.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-10T16:55:09.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-10T16:55:09.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:55:09.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:55:09.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:55:09.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:55:09.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:55:09.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:55:09.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:55:09.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:55:09.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:55:09.543 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:55:10.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:55:10.009 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:10.009 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:55:10.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:55:10.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:55:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-10T16:55:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-10T16:55:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-10T16:55:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.152 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:55:10.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673027 2024-09-10T16:55:10.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673027 2024-09-10T16:55:10.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027' 2024-09-10T16:55:10.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:55:10.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509502 2024-09-10T16:55:10.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509502 2024-09-10T16:55:10.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502' 2024-09-10T16:55:10.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:55:10.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345976 2024-09-10T16:55:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345976 2024-09-10T16:55:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976' 2024-09-10T16:55:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.610 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:55:10.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182450 2024-09-10T16:55:10.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182450 2024-09-10T16:55:10.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450' 2024-09-10T16:55:10.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.721 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:55:10.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018922 2024-09-10T16:55:10.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018922 2024-09-10T16:55:10.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922' 2024-09-10T16:55:10.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:55:10.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855395 2024-09-10T16:55:10.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855395 2024-09-10T16:55:10.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922 6-150323855395' 2024-09-10T16:55:10.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:10.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:55:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691866 2024-09-10T16:55:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691866 2024-09-10T16:55:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922 6-150323855395 7-171798691866' 2024-09-10T16:55:11.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:11.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:55:11.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528341 2024-09-10T16:55:11.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528341 2024-09-10T16:55:11.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922 6-150323855395 7-171798691866 8-193273528341' 2024-09-10T16:55:11.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:11.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:55:11.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364817 2024-09-10T16:55:11.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364817 2024-09-10T16:55:11.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922 6-150323855395 7-171798691866 8-193273528341 9-214748364817' 2024-09-10T16:55:11.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:11.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:55:11.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201292 2024-09-10T16:55:11.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201292 2024-09-10T16:55:11.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673027 2-64424509502 3-85899345976 4-107374182450 5-128849018922 6-150323855395 7-171798691866 8-193273528341 9-214748364817 10-236223201292' 2024-09-10T16:55:11.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:11.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-10T16:55:11.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:11.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:55:11.387 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-10T16:55:11.387 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:11.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-10T16:55:11.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-10T16:55:11.389 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836554 2024-09-10T16:55:11.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:55:11.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836554 2024-09-10T16:55:11.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T16:55:12.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T16:55:12.773 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:55:13.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836554 2024-09-10T16:55:13.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:13.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673027 2024-09-10T16:55:13.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:13.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:55:13.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673027 2024-09-10T16:55:13.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:13.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673027 2024-09-10T16:55:13.214 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949673027 2024-09-10T16:55:13.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-09-10T16:55:13.215 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:55:13.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673027 2024-09-10T16:55:13.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:13.594 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509502 2024-09-10T16:55:13.594 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:13.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:55:13.595 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509502 2024-09-10T16:55:13.595 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:13.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509502 2024-09-10T16:55:13.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-09-10T16:55:13.596 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509502 2024-09-10T16:55:13.597 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:55:14.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509502 2024-09-10T16:55:14.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:14.015 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345976 2024-09-10T16:55:14.015 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:14.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:55:14.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345976 2024-09-10T16:55:14.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:14.018 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 85899345976 2024-09-10T16:55:14.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345976 2024-09-10T16:55:14.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345976' 2024-09-10T16:55:14.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:55:14.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345977 -lt 85899345976 2024-09-10T16:55:14.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:14.433 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182450 2024-09-10T16:55:14.433 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:14.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:55:14.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182450 2024-09-10T16:55:14.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182450 2024-09-10T16:55:14.436 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 107374182450 2024-09-10T16:55:14.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-09-10T16:55:14.436 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:55:14.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182451 -lt 107374182450 2024-09-10T16:55:14.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:14.837 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018922 2024-09-10T16:55:14.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:14.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:55:14.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018922 2024-09-10T16:55:14.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:14.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018922 2024-09-10T16:55:14.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-09-10T16:55:14.840 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 128849018922 2024-09-10T16:55:14.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:55:15.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018922 -lt 128849018922 2024-09-10T16:55:15.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:15.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855395 2024-09-10T16:55:15.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:15.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:55:15.235 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855395 2024-09-10T16:55:15.235 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:15.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855395 2024-09-10T16:55:15.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-09-10T16:55:15.236 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 150323855395 2024-09-10T16:55:15.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:55:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855395 -lt 150323855395 2024-09-10T16:55:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:15.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691866 2024-09-10T16:55:15.663 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:15.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:55:15.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691866 2024-09-10T16:55:15.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:15.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691866 2024-09-10T16:55:15.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-09-10T16:55:15.665 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 171798691866 2024-09-10T16:55:15.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:55:16.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691867 -lt 171798691866 2024-09-10T16:55:16.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:16.064 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528341 2024-09-10T16:55:16.064 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:16.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:55:16.066 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528341 2024-09-10T16:55:16.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528341 2024-09-10T16:55:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-09-10T16:55:16.067 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 193273528341 2024-09-10T16:55:16.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:55:16.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528342 -lt 193273528341 2024-09-10T16:55:16.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:16.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364817 2024-09-10T16:55:16.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:16.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:55:16.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364817 2024-09-10T16:55:16.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:16.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364817 2024-09-10T16:55:16.493 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.9 seq 214748364817 2024-09-10T16:55:16.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-09-10T16:55:16.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:55:17.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364818 -lt 214748364817 2024-09-10T16:55:17.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:17.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201292 2024-09-10T16:55:17.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:17.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:55:17.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201292 2024-09-10T16:55:17.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:17.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201292 2024-09-10T16:55:17.013 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.10 seq 236223201292 2024-09-10T16:55:17.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201292' 2024-09-10T16:55:17.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:55:17.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201293 -lt 236223201292 2024-09-10T16:55:17.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:55:17.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:17.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:18.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-10T16:55:18.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:55:18.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:55:18.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:55:18.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:55:18.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:55:18.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:55:18.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:55:18.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T16:55:18.387 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:55:18.387 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:18.387 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:18.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-10T16:55:18.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:55:18.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:55:18.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55334/ceph-osd.0.asok 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-10T16:55:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.55334/ceph-osd.0.asok log flush 2024-09-10T16:55:18.985 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:55:18.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-10T16:55:18.996 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:48:51.609+0000 7fa4cad1e640 0 load: jerasure load: lrc 2024-09-10T16:55:18.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-10T16:55:18.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-10T16:55:18.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-10T16:55:19.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-10T16:55:19.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-10T16:55:20.329 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'ecpool' already exists 2024-09-10T16:55:20.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:55:21.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-10T16:55:21.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:55:21.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:55:21.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:55:21.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:55:21.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:55:22.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:22.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:55:22.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836559 2024-09-10T16:55:22.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836559 2024-09-10T16:55:22.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559' 2024-09-10T16:55:22.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:22.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:55:22.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-10T16:55:22.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-10T16:55:22.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031' 2024-09-10T16:55:22.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:22.702 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:55:23.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509506 2024-09-10T16:55:23.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509506 2024-09-10T16:55:23.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506' 2024-09-10T16:55:23.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:23.152 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:55:23.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345981 2024-09-10T16:55:23.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345981 2024-09-10T16:55:23.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981' 2024-09-10T16:55:23.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:23.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:55:23.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182455 2024-09-10T16:55:23.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182455 2024-09-10T16:55:23.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455' 2024-09-10T16:55:23.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:23.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:55:23.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018926 2024-09-10T16:55:23.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018926 2024-09-10T16:55:23.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926' 2024-09-10T16:55:23.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:23.902 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:55:24.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855399 2024-09-10T16:55:24.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855399 2024-09-10T16:55:24.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926 6-150323855399' 2024-09-10T16:55:24.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:24.065 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:55:24.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691871 2024-09-10T16:55:24.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691871 2024-09-10T16:55:24.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926 6-150323855399 7-171798691871' 2024-09-10T16:55:24.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:24.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:55:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528346 2024-09-10T16:55:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528346 2024-09-10T16:55:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926 6-150323855399 7-171798691871 8-193273528346' 2024-09-10T16:55:24.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:24.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:55:24.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364822 2024-09-10T16:55:24.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364822 2024-09-10T16:55:24.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926 6-150323855399 7-171798691871 8-193273528346 9-214748364822' 2024-09-10T16:55:24.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:24.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:55:24.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201297 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201297 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673031 2-64424509506 3-85899345981 4-107374182455 5-128849018926 6-150323855399 7-171798691871 8-193273528346 9-214748364822 10-236223201297' 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836559 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:24.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:55:24.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836559 2024-09-10T16:55:24.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:24.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836559 2024-09-10T16:55:24.634 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836559 2024-09-10T16:55:24.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-10T16:55:24.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:55:25.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-10T16:55:25.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:25.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-10T16:55:25.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:25.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:55:25.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-10T16:55:25.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:25.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-10T16:55:25.070 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949673031 2024-09-10T16:55:25.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-10T16:55:25.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:55:25.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-10T16:55:25.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:25.547 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509506 2024-09-10T16:55:25.548 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:25.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:55:25.549 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509506 2024-09-10T16:55:25.551 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:25.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509506 2024-09-10T16:55:25.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-09-10T16:55:25.551 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509506 2024-09-10T16:55:25.551 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:55:25.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509507 -lt 64424509506 2024-09-10T16:55:25.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:25.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345981 2024-09-10T16:55:25.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:25.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:55:25.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345981 2024-09-10T16:55:25.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:25.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345981 2024-09-10T16:55:25.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345981' 2024-09-10T16:55:25.937 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 85899345981 2024-09-10T16:55:25.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:55:26.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345981 -lt 85899345981 2024-09-10T16:55:26.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:26.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182455 2024-09-10T16:55:26.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:26.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:55:26.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182455 2024-09-10T16:55:26.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:26.439 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 107374182455 2024-09-10T16:55:26.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182455 2024-09-10T16:55:26.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-09-10T16:55:26.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:55:26.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182455 -lt 107374182455 2024-09-10T16:55:26.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:26.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018926 2024-09-10T16:55:26.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:26.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:55:26.820 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018926 2024-09-10T16:55:26.820 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:26.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018926 2024-09-10T16:55:26.821 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 128849018926 2024-09-10T16:55:26.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-09-10T16:55:26.822 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:55:27.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018927 -lt 128849018926 2024-09-10T16:55:27.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:27.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855399 2024-09-10T16:55:27.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:27.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:55:27.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855399 2024-09-10T16:55:27.206 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:27.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855399 2024-09-10T16:55:27.207 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 150323855399 2024-09-10T16:55:27.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-09-10T16:55:27.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:55:27.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855400 -lt 150323855399 2024-09-10T16:55:27.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:27.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691871 2024-09-10T16:55:27.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:27.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:55:27.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691871 2024-09-10T16:55:27.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:27.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691871 2024-09-10T16:55:27.592 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 171798691871 2024-09-10T16:55:27.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-09-10T16:55:27.592 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:55:27.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691872 -lt 171798691871 2024-09-10T16:55:27.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:27.975 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528346 2024-09-10T16:55:27.975 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:27.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:55:27.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528346 2024-09-10T16:55:27.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:27.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528346 2024-09-10T16:55:27.978 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 193273528346 2024-09-10T16:55:27.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-09-10T16:55:27.978 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:55:28.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528347 -lt 193273528346 2024-09-10T16:55:28.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:28.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364822 2024-09-10T16:55:28.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:28.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:55:28.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364822 2024-09-10T16:55:28.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:28.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364822 2024-09-10T16:55:28.367 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.9 seq 214748364822 2024-09-10T16:55:28.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364822' 2024-09-10T16:55:28.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:55:28.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364822 -lt 214748364822 2024-09-10T16:55:28.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:28.789 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201297 2024-09-10T16:55:28.789 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:28.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:55:28.791 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201297 2024-09-10T16:55:28.791 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:28.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201297 2024-09-10T16:55:28.792 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.10 seq 236223201297 2024-09-10T16:55:28.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201297' 2024-09-10T16:55:28.792 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:55:29.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201297 -lt 236223201297 2024-09-10T16:55:29.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:55:29.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:29.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:29.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T16:55:29.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:55:29.696 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:55:29.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:55:29.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:55:29.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:55:29.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:55:29.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:55:30.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T16:55:30.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:55:30.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:30.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:30.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T16:55:30.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:55:30.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:55:30.640 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-10T16:55:30.640 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T16:55:30.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_chunk_mapping 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_put_get_isa 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_put_get_jerasure 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_put_get_lrc_advanced 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_put_get_lrc_kml 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_put_get_shec' 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-09-10T16:55:30.642 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-10T16:55:30.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-09-10T16:55:30.658 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-10T16:55:30.658 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-09-10T16:55:31.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:2+0 records in 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:2+0 records out 2024-09-10T16:55:31.075 INFO:tasks.workunit.client.0.smithi086.stderr:16382 bytes (16 kB, 16 KiB) copied, 8.9361e-05 s, 183 MB/s 2024-09-10T16:55:31.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-10T16:55:31.104 INFO:tasks.workunit.client.0.smithi086.stderr:INFO: op_size has been rounded to 8192 2024-09-10T16:55:31.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:55:31.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-09-10T16:55:31.474 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-10T16:55:31.475 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-10T16:55:31.490 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-10T16:55:31.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-10T16:55:31.491 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-10T16:55:31.491 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-10T16:55:31.507 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-10T16:55:31.507 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-10T16:55:31.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-10T16:55:31.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-10T16:55:31.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-10T16:55:31.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-10T16:55:31.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-10T16:55:31.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T16:55:31.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-10T16:55:31.825 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-10T16:55:31.825 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:55:32.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T16:55:32.225 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:32.225 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:0' 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-10T16:55:32.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-10T16:55:32.318 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:55:32.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:55:32.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-10T16:55:32.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-10T16:55:32.413 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:55:32.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:55:32.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-10T16:55:32.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-10T16:55:32.509 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:55:32.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:55:32.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-10T16:55:32.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-10T16:55:32.604 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:55:32.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:55:32.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-09-10T16:55:32.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:55:32.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-10T16:55:32.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-10T16:55:32.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T16:55:32.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2024-09-10T16:55:32.608 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-10T16:55:32.608 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:0' 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-10T16:55:33.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-10T16:55:33.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:55:33.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:55:33.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:55:33.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:55:33.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:55:33.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:55:33.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-10T16:55:33.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-10T16:55:33.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T16:55:33.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T16:55:33.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T16:55:33.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-10T16:55:33.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T16:55:34.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:55:34.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:55:34.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.3 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-10T16:55:34.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T16:55:34.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T16:55:34.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T16:55:34.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T16:55:34.258 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:34.256+0000 7faba241e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:34.260 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:34.258+0000 7faba241e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:34.261 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:34.260+0000 7faba241e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:34.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:34.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:55:35.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:35.874 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:35.873+0000 7faba241e640 -1 Falling back to public interface 2024-09-10T16:55:36.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:36.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:36.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:55:36.110 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:36.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:36.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:55:36.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:37.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:55:37.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:38.512 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:38.511+0000 7faba241e640 -1 osd.3 77 log_to_monitors true 2024-09-10T16:55:38.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:38.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:38.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:55:38.933 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:55:38.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:38.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T16:55:39.407 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 up in weight 1 up_from 81 up_thru 81 down_at 78 last_clean_interval [20,77) [v2:127.0.0.1:6826/1923561070,v1:127.0.0.1:6827/1923561070] [v2:127.0.0.1:6828/1923561070,v1:127.0.0.1:6829/1923561070] exists,up 24dc7c4c-ac60-4a9b-a786-0215296fb72d 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:55:39.408 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:55:39.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:55:39.490 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:55:40.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:55:40.319 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:40.319 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:40.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:55:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836564 2024-09-10T16:55:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836564 2024-09-10T16:55:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564' 2024-09-10T16:55:40.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:40.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:55:40.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673037 2024-09-10T16:55:40.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673037 2024-09-10T16:55:40.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037' 2024-09-10T16:55:40.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:40.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:55:40.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-10T16:55:40.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-10T16:55:40.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512' 2024-09-10T16:55:40.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:40.898 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:55:41.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350979 2024-09-10T16:55:41.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350979 2024-09-10T16:55:41.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979' 2024-09-10T16:55:41.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:55:41.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182461 2024-09-10T16:55:41.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182461 2024-09-10T16:55:41.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461' 2024-09-10T16:55:41.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:55:41.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018932 2024-09-10T16:55:41.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018932 2024-09-10T16:55:41.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932' 2024-09-10T16:55:41.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:55:41.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855405 2024-09-10T16:55:41.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855405 2024-09-10T16:55:41.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932 6-150323855405' 2024-09-10T16:55:41.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:55:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691876 2024-09-10T16:55:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691876 2024-09-10T16:55:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932 6-150323855405 7-171798691876' 2024-09-10T16:55:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:55:41.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528351 2024-09-10T16:55:41.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528351 2024-09-10T16:55:41.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932 6-150323855405 7-171798691876 8-193273528351' 2024-09-10T16:55:41.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:55:41.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364827 2024-09-10T16:55:41.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364827 2024-09-10T16:55:41.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932 6-150323855405 7-171798691876 8-193273528351 9-214748364827' 2024-09-10T16:55:41.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:41.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:55:41.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201302 2024-09-10T16:55:41.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201302 2024-09-10T16:55:41.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509512 3-347892350979 4-107374182461 5-128849018932 6-150323855405 7-171798691876 8-193273528351 9-214748364827 10-236223201302' 2024-09-10T16:55:41.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:41.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836564 2024-09-10T16:55:41.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:41.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:55:41.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836564 2024-09-10T16:55:41.800 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:41.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836564 2024-09-10T16:55:41.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-09-10T16:55:41.800 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 21474836564 2024-09-10T16:55:41.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:55:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836564 2024-09-10T16:55:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:42.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673037 2024-09-10T16:55:42.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:42.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:55:42.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673037 2024-09-10T16:55:42.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673037 2024-09-10T16:55:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-09-10T16:55:42.268 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 42949673037 2024-09-10T16:55:42.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:55:42.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673037 -lt 42949673037 2024-09-10T16:55:42.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:42.653 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-10T16:55:42.653 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:42.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:55:42.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-10T16:55:42.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:42.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-10T16:55:42.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-10T16:55:42.655 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 64424509512 2024-09-10T16:55:42.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:55:43.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-10T16:55:43.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:43.077 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350979 2024-09-10T16:55:43.077 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:43.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:55:43.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350979 2024-09-10T16:55:43.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:43.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350979 2024-09-10T16:55:43.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350979' 2024-09-10T16:55:43.080 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 347892350979 2024-09-10T16:55:43.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:55:43.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350979 -lt 347892350979 2024-09-10T16:55:43.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:43.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182461 2024-09-10T16:55:43.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:43.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:55:43.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182461 2024-09-10T16:55:43.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:43.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182461 2024-09-10T16:55:43.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-09-10T16:55:43.462 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.4 seq 107374182461 2024-09-10T16:55:43.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:55:43.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182461 -lt 107374182461 2024-09-10T16:55:43.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:43.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018932 2024-09-10T16:55:43.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:43.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:55:43.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018932 2024-09-10T16:55:43.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:43.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018932 2024-09-10T16:55:43.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018932' 2024-09-10T16:55:43.874 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.5 seq 128849018932 2024-09-10T16:55:43.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:55:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018932 -lt 128849018932 2024-09-10T16:55:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:44.259 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855405 2024-09-10T16:55:44.259 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:55:44.260 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855405 2024-09-10T16:55:44.260 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855405 2024-09-10T16:55:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-09-10T16:55:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.6 seq 150323855405 2024-09-10T16:55:44.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:55:44.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855405 -lt 150323855405 2024-09-10T16:55:44.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:44.684 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691876 2024-09-10T16:55:44.684 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:44.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:55:44.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691876 2024-09-10T16:55:44.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:44.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691876 2024-09-10T16:55:44.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691876' 2024-09-10T16:55:44.686 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.7 seq 171798691876 2024-09-10T16:55:44.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:55:45.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691877 -lt 171798691876 2024-09-10T16:55:45.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:45.110 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528351 2024-09-10T16:55:45.110 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:45.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:55:45.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528351 2024-09-10T16:55:45.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:45.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528351 2024-09-10T16:55:45.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-09-10T16:55:45.113 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.8 seq 193273528351 2024-09-10T16:55:45.113 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:55:45.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528352 -lt 193273528351 2024-09-10T16:55:45.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:45.493 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364827 2024-09-10T16:55:45.493 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:45.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:55:45.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364827 2024-09-10T16:55:45.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:45.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364827 2024-09-10T16:55:45.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364827' 2024-09-10T16:55:45.495 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.9 seq 214748364827 2024-09-10T16:55:45.495 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:55:45.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364828 -lt 214748364827 2024-09-10T16:55:45.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:45.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201302 2024-09-10T16:55:45.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:45.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:55:45.929 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201302 2024-09-10T16:55:45.930 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:45.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201302 2024-09-10T16:55:45.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201302' 2024-09-10T16:55:45.931 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.10 seq 236223201302 2024-09-10T16:55:45.931 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:55:46.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201303 -lt 236223201302 2024-09-10T16:55:46.373 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:55:46.374 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:46.374 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:55:46.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:55:46.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:55:46.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:55:47.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T16:55:47.304 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:55:47.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:55:47.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:55:47.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T16:55:47.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:55:47.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:55:47.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-10T16:55:47.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-10T16:55:47.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T16:55:47.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:55:47.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T16:55:48.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-10T16:55:48.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-10T16:55:48.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-10T16:55:48.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-10T16:55:48.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T16:55:48.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T16:55:48.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:55:48.907 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T16:55:48.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.5 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-10T16:55:48.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T16:55:48.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T16:55:48.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T16:55:48.931 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:48.929+0000 7fd28529c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:49.217 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:49.215+0000 7fd28529c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:49.218 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:49.216+0000 7fd28529c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:55:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-10T16:55:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:55:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T16:55:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:55:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:55:49.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:49.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:55:49.685 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:55:49.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:49.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:55:50.073 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:50.072+0000 7fd28529c640 -1 Falling back to public interface 2024-09-10T16:55:50.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:51.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:51.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:51.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:55:51.205 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:51.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:51.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:55:51.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:55:51.949 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:55:51.947+0000 7fd28529c640 -1 osd.5 83 log_to_monitors true 2024-09-10T16:55:52.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:55:52.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:55:52.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:55:52.593 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:52.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:55:52.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:osd.5 up in weight 1 up_from 88 up_thru 88 down_at 84 last_clean_interval [30,83) [v2:127.0.0.1:6842/3054244253,v1:127.0.0.1:6843/3054244253] [v2:127.0.0.1:6844/3054244253,v1:127.0.0.1:6845/3054244253] exists,up 50cee0f8-81be-4ca4-afd9-8355fe76a6ff 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:55:53.337 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:55:53.338 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:55:53.338 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:55:53.338 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:55:53.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:55:53.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:55:53.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:55:53.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:55:53.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:55:53.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:55:54.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:55:54.396 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:55:54.397 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:55:54.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:55:54.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:54.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:55:54.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-10T16:55:54.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-10T16:55:54.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-10T16:55:54.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:54.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:55:54.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673042 2024-09-10T16:55:54.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673042 2024-09-10T16:55:54.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042' 2024-09-10T16:55:54.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:54.760 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:55:54.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509517 2024-09-10T16:55:54.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509517 2024-09-10T16:55:54.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517' 2024-09-10T16:55:54.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:54.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:55:55.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350984 2024-09-10T16:55:55.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350984 2024-09-10T16:55:55.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984' 2024-09-10T16:55:55.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:55:55.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182465 2024-09-10T16:55:55.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182465 2024-09-10T16:55:55.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465' 2024-09-10T16:55:55.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:55:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122051 2024-09-10T16:55:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122051 2024-09-10T16:55:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051' 2024-09-10T16:55:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:55:55.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855410 2024-09-10T16:55:55.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855410 2024-09-10T16:55:55.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051 6-150323855410' 2024-09-10T16:55:55.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:55:55.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691881 2024-09-10T16:55:55.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691881 2024-09-10T16:55:55.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051 6-150323855410 7-171798691881' 2024-09-10T16:55:55.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.493 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:55:55.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528356 2024-09-10T16:55:55.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528356 2024-09-10T16:55:55.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051 6-150323855410 7-171798691881 8-193273528356' 2024-09-10T16:55:55.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.602 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:55:55.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364832 2024-09-10T16:55:55.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364832 2024-09-10T16:55:55.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051 6-150323855410 7-171798691881 8-193273528356 9-214748364832' 2024-09-10T16:55:55.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:55:55.716 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:55:55.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201307 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201307 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673042 2-64424509517 3-347892350984 4-107374182465 5-377957122051 6-150323855410 7-171798691881 8-193273528356 9-214748364832 10-236223201307' 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:55.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:55:55.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-10T16:55:55.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:55.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-10T16:55:55.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-10T16:55:55.832 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 21474836569 2024-09-10T16:55:55.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:55:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-10T16:55:56.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:56.327 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673042 2024-09-10T16:55:56.327 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:56.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:55:56.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673042 2024-09-10T16:55:56.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:56.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673042 2024-09-10T16:55:56.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-09-10T16:55:56.329 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 42949673042 2024-09-10T16:55:56.329 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:55:56.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673042 -lt 42949673042 2024-09-10T16:55:56.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:56.712 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509517 2024-09-10T16:55:56.712 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:56.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:55:56.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509517 2024-09-10T16:55:56.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:56.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509517 2024-09-10T16:55:56.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-09-10T16:55:56.715 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 64424509517 2024-09-10T16:55:56.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:55:57.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509517 2024-09-10T16:55:57.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:57.100 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350984 2024-09-10T16:55:57.100 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:57.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:55:57.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350984 2024-09-10T16:55:57.102 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:57.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350984 2024-09-10T16:55:57.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350984' 2024-09-10T16:55:57.103 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 347892350984 2024-09-10T16:55:57.103 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:55:57.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350984 -lt 347892350984 2024-09-10T16:55:57.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:57.524 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182465 2024-09-10T16:55:57.524 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:57.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:55:57.525 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182465 2024-09-10T16:55:57.525 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:57.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182465 2024-09-10T16:55:57.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-09-10T16:55:57.526 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.4 seq 107374182465 2024-09-10T16:55:57.526 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:55:57.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182465 -lt 107374182465 2024-09-10T16:55:57.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:57.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122051 2024-09-10T16:55:57.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:57.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:55:57.910 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122051 2024-09-10T16:55:57.910 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:57.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122051 2024-09-10T16:55:57.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122051' 2024-09-10T16:55:57.911 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.5 seq 377957122051 2024-09-10T16:55:57.911 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:55:58.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122052 -lt 377957122051 2024-09-10T16:55:58.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:58.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855410 2024-09-10T16:55:58.335 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:58.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:55:58.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855410 2024-09-10T16:55:58.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:58.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855410 2024-09-10T16:55:58.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-09-10T16:55:58.338 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.6 seq 150323855410 2024-09-10T16:55:58.338 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:55:58.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855410 -lt 150323855410 2024-09-10T16:55:58.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:58.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691881 2024-09-10T16:55:58.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:58.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:55:58.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691881 2024-09-10T16:55:58.755 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:58.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691881 2024-09-10T16:55:58.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-09-10T16:55:58.756 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.7 seq 171798691881 2024-09-10T16:55:58.756 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:55:59.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691882 -lt 171798691881 2024-09-10T16:55:59.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:59.172 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528356 2024-09-10T16:55:59.172 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:59.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:55:59.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528356 2024-09-10T16:55:59.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:59.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528356 2024-09-10T16:55:59.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-09-10T16:55:59.174 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.8 seq 193273528356 2024-09-10T16:55:59.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:55:59.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528357 -lt 193273528356 2024-09-10T16:55:59.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:59.598 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364832 2024-09-10T16:55:59.598 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:59.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:55:59.599 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364832 2024-09-10T16:55:59.599 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:59.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364832 2024-09-10T16:55:59.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364832' 2024-09-10T16:55:59.601 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.9 seq 214748364832 2024-09-10T16:55:59.601 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:55:59.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364833 -lt 214748364832 2024-09-10T16:55:59.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:55:59.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201307 2024-09-10T16:55:59.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:55:59.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:55:59.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201307 2024-09-10T16:55:59.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:55:59.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201307 2024-09-10T16:55:59.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201307' 2024-09-10T16:55:59.984 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.10 seq 236223201307 2024-09-10T16:55:59.984 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:56:00.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201308 -lt 236223201307 2024-09-10T16:56:00.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:56:00.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:00.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:56:00.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:56:01.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T16:56:01.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:56:01.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:01.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:01.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T16:56:01.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:56:01.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:56:01.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-10T16:56:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-10T16:56:02.844 INFO:tasks.workunit.client.0.smithi086.stdout:crush-device-class= 2024-09-10T16:56:02.844 INFO:tasks.workunit.client.0.smithi086.stdout:crush-failure-domain=host 2024-09-10T16:56:02.844 INFO:tasks.workunit.client.0.smithi086.stdout:crush-num-failure-domains=0 2024-09-10T16:56:02.844 INFO:tasks.workunit.client.0.smithi086.stdout:crush-osds-per-failure-domain=0 2024-09-10T16:56:02.844 INFO:tasks.workunit.client.0.smithi086.stdout:crush-root=default 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:k=-1 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:l=-1 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:layers=[ [ "cDD", "" ] ] 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:m=-1 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:mapping=_DD 2024-09-10T16:56:02.845 INFO:tasks.workunit.client.0.smithi086.stdout:plugin=lrc 2024-09-10T16:56:02.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-10T16:56:02.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-10T16:56:03.778 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'remap-pool' already exists 2024-09-10T16:56:03.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-09-10T16:56:04.792 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-10T16:56:04.793 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-10T16:56:04.810 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-10T16:56:04.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-10T16:56:04.810 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-09-10T16:56:04.810 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-10T16:56:04.827 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-09-10T16:56:04.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-10T16:56:04.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-10T16:56:04.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-10T16:56:04.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-10T16:56:06.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-10T16:56:06.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-10T16:56:06.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-10T16:56:06.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-10T16:56:06.353 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-10T16:56:06.353 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:56:07.125 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-10T16:56:07.125 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:07.125 INFO:tasks.workunit.client.0.smithi086.stderr:9' 2024-09-10T16:56:07.125 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-10T16:56:07.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-10T16:56:07.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-09-10T16:56:07.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-09-10T16:56:07.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-10T16:56:07.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-10T16:56:07.218 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:56:07.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:56:07.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-10T16:56:07.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-10T16:56:07.315 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:56:07.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:56:07.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-10T16:56:07.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-10T16:56:07.410 INFO:tasks.workunit.client.0.smithi086.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-10T16:56:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-09-10T16:56:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-09-10T16:56:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:56:07.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-10T16:56:07.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-10T16:56:07.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2024-09-10T16:56:07.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2024-09-10T16:56:07.414 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-10T16:56:07.414 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:56:07.841 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-10T16:56:07.841 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:07.841 INFO:tasks.workunit.client.0.smithi086.stderr:9' 2024-09-10T16:56:07.841 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=6 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=6 2024-09-10T16:56:07.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:56:07.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:56:09.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:56:09.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:09.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-10T16:56:09.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T16:56:09.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=6 2024-09-10T16:56:09.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T16:56:09.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-10T16:56:09.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:09.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-10T16:56:09.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=6 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T16:56:09.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:56:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.6 2024-09-10T16:56:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.6 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 6 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 6 = 6 ']' 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T16:56:09.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T16:56:09.976 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:09.974+0000 7fccc0887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:09.980 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:09.978+0000 7fccc0887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:09.981 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:09.980+0000 7fccc0887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 6 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:56:10.403 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:56:10.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:10.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:56:10.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:11.335 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:11.334+0000 7fccc0887640 -1 Falling back to public interface 2024-09-10T16:56:11.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:11.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:11.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:56:11.822 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:11.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:11.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:56:12.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:13.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:13.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:13.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:56:13.237 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:56:13.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:13.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:56:13.535 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:13.533+0000 7fccc0887640 -1 osd.6 99 log_to_monitors true 2024-09-10T16:56:13.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:14.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:14.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:14.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:56:14.624 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:56:14.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:14.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T16:56:15.618 INFO:tasks.workunit.client.0.smithi086.stderr:osd.6 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [35,99) [v2:127.0.0.1:6850/3410569944,v1:127.0.0.1:6851/3410569944] [v2:127.0.0.1:6852/3410569944,v1:127.0.0.1:6853/3410569944] exists,up 36b78b87-c9dd-4080-8317-c51c3be1fc24 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:56:15.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:56:15.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:56:15.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:56:15.701 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:56:16.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:16.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:56:16.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836576 2024-09-10T16:56:16.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836576 2024-09-10T16:56:16.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576' 2024-09-10T16:56:16.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:16.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:56:17.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673048 2024-09-10T16:56:17.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673048 2024-09-10T16:56:17.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048' 2024-09-10T16:56:17.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.095 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:56:17.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-10T16:56:17.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-10T16:56:17.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523' 2024-09-10T16:56:17.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.208 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:56:17.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350990 2024-09-10T16:56:17.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350990 2024-09-10T16:56:17.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990' 2024-09-10T16:56:17.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:56:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182472 2024-09-10T16:56:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182472 2024-09-10T16:56:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472' 2024-09-10T16:56:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:56:17.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122058 2024-09-10T16:56:17.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122058 2024-09-10T16:56:17.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058' 2024-09-10T16:56:17.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:56:17.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-10T16:56:17.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-10T16:56:17.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058 6-455266533379' 2024-09-10T16:56:17.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:56:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691888 2024-09-10T16:56:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691888 2024-09-10T16:56:17.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058 6-455266533379 7-171798691888' 2024-09-10T16:56:17.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:56:17.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528363 2024-09-10T16:56:17.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528363 2024-09-10T16:56:17.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058 6-455266533379 7-171798691888 8-193273528363' 2024-09-10T16:56:17.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:17.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:56:18.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364838 2024-09-10T16:56:18.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364838 2024-09-10T16:56:18.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058 6-455266533379 7-171798691888 8-193273528363 9-214748364838' 2024-09-10T16:56:18.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:18.001 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201313 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201313 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673048 2-64424509523 3-347892350990 4-107374182472 5-377957122058 6-455266533379 7-171798691888 8-193273528363 9-214748364838 10-236223201313' 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836576 2024-09-10T16:56:18.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:18.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:56:18.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836576 2024-09-10T16:56:18.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:18.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836576 2024-09-10T16:56:18.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-09-10T16:56:18.116 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 21474836576 2024-09-10T16:56:18.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:56:18.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836576 -lt 21474836576 2024-09-10T16:56:18.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:18.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673048 2024-09-10T16:56:18.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:18.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:56:18.533 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673048 2024-09-10T16:56:18.533 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:18.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673048 2024-09-10T16:56:18.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-09-10T16:56:18.534 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 42949673048 2024-09-10T16:56:18.534 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:56:18.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673048 2024-09-10T16:56:18.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:18.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-10T16:56:18.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:18.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:56:18.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-10T16:56:18.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:18.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-10T16:56:18.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-10T16:56:18.978 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 64424509523 2024-09-10T16:56:18.978 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:56:19.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509523 -lt 64424509523 2024-09-10T16:56:19.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:19.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350990 2024-09-10T16:56:19.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:19.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:56:19.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350990 2024-09-10T16:56:19.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:19.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350990 2024-09-10T16:56:19.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350990' 2024-09-10T16:56:19.463 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 347892350990 2024-09-10T16:56:19.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:56:19.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350990 -lt 347892350990 2024-09-10T16:56:19.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:19.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182472 2024-09-10T16:56:19.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:19.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:56:19.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182472 2024-09-10T16:56:19.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:19.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182472 2024-09-10T16:56:19.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-09-10T16:56:19.882 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.4 seq 107374182472 2024-09-10T16:56:19.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:56:20.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182472 -lt 107374182472 2024-09-10T16:56:20.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:20.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122058 2024-09-10T16:56:20.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:20.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:56:20.324 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122058 2024-09-10T16:56:20.324 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:20.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122058 2024-09-10T16:56:20.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122058' 2024-09-10T16:56:20.325 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.5 seq 377957122058 2024-09-10T16:56:20.325 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:56:20.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122058 -lt 377957122058 2024-09-10T16:56:20.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:20.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533379 2024-09-10T16:56:20.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:20.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:56:20.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533379 2024-09-10T16:56:20.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:20.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-10T16:56:20.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533379' 2024-09-10T16:56:20.752 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.6 seq 455266533379 2024-09-10T16:56:20.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:56:21.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-10T16:56:21.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:21.176 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691888 2024-09-10T16:56:21.176 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:21.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:56:21.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691888 2024-09-10T16:56:21.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:21.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691888 2024-09-10T16:56:21.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-09-10T16:56:21.178 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.7 seq 171798691888 2024-09-10T16:56:21.178 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:56:21.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691888 -lt 171798691888 2024-09-10T16:56:21.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:21.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528363 2024-09-10T16:56:21.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:21.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:56:21.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528363 2024-09-10T16:56:21.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:21.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528363 2024-09-10T16:56:21.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-09-10T16:56:21.568 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.8 seq 193273528363 2024-09-10T16:56:21.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:56:21.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528364 -lt 193273528363 2024-09-10T16:56:21.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:21.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364838 2024-09-10T16:56:21.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:21.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:56:21.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364838 2024-09-10T16:56:21.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:21.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364838 2024-09-10T16:56:21.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364838' 2024-09-10T16:56:21.971 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.9 seq 214748364838 2024-09-10T16:56:21.971 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:56:22.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364839 -lt 214748364838 2024-09-10T16:56:22.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:22.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201313 2024-09-10T16:56:22.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:22.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:56:22.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201313 2024-09-10T16:56:22.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:22.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201313 2024-09-10T16:56:22.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201313' 2024-09-10T16:56:22.393 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.10 seq 236223201313 2024-09-10T16:56:22.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:56:22.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201314 -lt 236223201313 2024-09-10T16:56:22.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:56:22.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:22.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:23.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-10T16:56:23.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:56:23.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:56:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-10T16:56:23.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:56:23.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:23.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:24.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-10T16:56:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:56:24.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:56:24.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-10T16:56:24.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T16:56:25.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:56:25.521 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T16:56:25.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.9 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=9905e8e1-d2f4-4d40-8eed-5463db93522b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55334/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T16:56:25.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T16:56:25.546 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:25.544+0000 7f889ef25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:25.548 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:25.546+0000 7f889ef25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:25.549 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:25.547+0000 7f889ef25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:56:26.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:26.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:56:26.263 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:56:26.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:26.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:56:26.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:26.898 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:26.897+0000 7f889ef25640 -1 Falling back to public interface 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:56:28.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:29.113 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:56:29.111+0000 7f889ef25640 -1 osd.9 109 log_to_monitors true 2024-09-10T16:56:29.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:29.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:29.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:56:29.222 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:56:29.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:29.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:56:29.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:56:30.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:56:30.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:56:30.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:56:30.671 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:56:30.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:56:30.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-10T16:56:31.406 INFO:tasks.workunit.client.0.smithi086.stderr:osd.9 up in weight 1 up_from 114 up_thru 114 down_at 110 last_clean_interval [50,109) [v2:127.0.0.1:6874/4093880724,v1:127.0.0.1:6875/4093880724] [v2:127.0.0.1:6876/4093880724,v1:127.0.0.1:6877/4093880724] exists,up a85f522e-e117-40bd-8988-e4dd30d5ffcf 2024-09-10T16:56:31.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:56:31.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:56:31.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:56:31.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:56:31.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:56:31.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:56:31.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:56:31.488 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:56:32.560 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:56:32.561 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:56:32.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:56:32.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:32.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:56:32.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836581 2024-09-10T16:56:32.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836581 2024-09-10T16:56:32.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581' 2024-09-10T16:56:32.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:32.672 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:56:32.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673053 2024-09-10T16:56:32.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673053 2024-09-10T16:56:32.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053' 2024-09-10T16:56:32.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:32.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:56:32.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509528 2024-09-10T16:56:32.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509528 2024-09-10T16:56:32.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528' 2024-09-10T16:56:32.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:32.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:56:33.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350995 2024-09-10T16:56:33.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350995 2024-09-10T16:56:33.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995' 2024-09-10T16:56:33.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.059 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:56:33.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182477 2024-09-10T16:56:33.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182477 2024-09-10T16:56:33.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477' 2024-09-10T16:56:33.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:56:33.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122063 2024-09-10T16:56:33.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122063 2024-09-10T16:56:33.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063' 2024-09-10T16:56:33.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:56:33.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533384 2024-09-10T16:56:33.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533384 2024-09-10T16:56:33.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063 6-455266533384' 2024-09-10T16:56:33.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:56:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691893 2024-09-10T16:56:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691893 2024-09-10T16:56:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063 6-455266533384 7-171798691893' 2024-09-10T16:56:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:56:33.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528368 2024-09-10T16:56:33.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528368 2024-09-10T16:56:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063 6-455266533384 7-171798691893 8-193273528368' 2024-09-10T16:56:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:56:33.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271747 2024-09-10T16:56:33.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271747 2024-09-10T16:56:33.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063 6-455266533384 7-171798691893 8-193273528368 9-489626271747' 2024-09-10T16:56:33.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:56:33.726 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:56:33.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201318 2024-09-10T16:56:33.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201318 2024-09-10T16:56:33.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673053 2-64424509528 3-347892350995 4-107374182477 5-377957122063 6-455266533384 7-171798691893 8-193273528368 9-489626271747 10-236223201318' 2024-09-10T16:56:33.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:33.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836581 2024-09-10T16:56:33.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:33.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:56:33.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836581 2024-09-10T16:56:33.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:33.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836581 2024-09-10T16:56:33.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-10T16:56:33.840 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 21474836581 2024-09-10T16:56:33.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:56:34.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-10T16:56:34.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:34.241 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673053 2024-09-10T16:56:34.242 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:34.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:56:34.242 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673053 2024-09-10T16:56:34.242 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:34.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673053 2024-09-10T16:56:34.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-09-10T16:56:34.244 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 42949673053 2024-09-10T16:56:34.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:56:34.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673053 -lt 42949673053 2024-09-10T16:56:34.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:34.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509528 2024-09-10T16:56:34.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:34.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:56:34.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509528 2024-09-10T16:56:34.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:34.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509528 2024-09-10T16:56:34.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-10T16:56:34.687 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 64424509528 2024-09-10T16:56:34.688 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:56:35.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509528 2024-09-10T16:56:35.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:35.072 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892350995 2024-09-10T16:56:35.072 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:35.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:56:35.073 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892350995 2024-09-10T16:56:35.073 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:35.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350995 2024-09-10T16:56:35.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892350995' 2024-09-10T16:56:35.074 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 347892350995 2024-09-10T16:56:35.075 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:56:35.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350996 -lt 347892350995 2024-09-10T16:56:35.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182477 2024-09-10T16:56:35.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:35.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:56:35.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182477 2024-09-10T16:56:35.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:35.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182477 2024-09-10T16:56:35.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-09-10T16:56:35.506 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.4 seq 107374182477 2024-09-10T16:56:35.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:56:35.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182477 -lt 107374182477 2024-09-10T16:56:35.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:35.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122063 2024-09-10T16:56:35.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:35.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:56:35.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122063 2024-09-10T16:56:35.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:35.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122063 2024-09-10T16:56:35.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122063' 2024-09-10T16:56:35.950 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.5 seq 377957122063 2024-09-10T16:56:35.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:56:36.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122063 -lt 377957122063 2024-09-10T16:56:36.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:36.401 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533384 2024-09-10T16:56:36.401 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:36.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:56:36.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533384 2024-09-10T16:56:36.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:36.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533384 2024-09-10T16:56:36.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533384' 2024-09-10T16:56:36.404 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.6 seq 455266533384 2024-09-10T16:56:36.404 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:56:36.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533385 -lt 455266533384 2024-09-10T16:56:36.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:36.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691893 2024-09-10T16:56:36.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:36.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:56:36.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691893 2024-09-10T16:56:36.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:36.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691893 2024-09-10T16:56:36.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-09-10T16:56:36.830 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.7 seq 171798691893 2024-09-10T16:56:36.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:56:37.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691893 -lt 171798691893 2024-09-10T16:56:37.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:37.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528368 2024-09-10T16:56:37.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:37.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:56:37.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528368 2024-09-10T16:56:37.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:37.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528368 2024-09-10T16:56:37.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-09-10T16:56:37.348 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.8 seq 193273528368 2024-09-10T16:56:37.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:56:37.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528368 -lt 193273528368 2024-09-10T16:56:37.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:37.765 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-489626271747 2024-09-10T16:56:37.765 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:37.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:56:37.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-489626271747 2024-09-10T16:56:37.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:37.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271747 2024-09-10T16:56:37.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 489626271747' 2024-09-10T16:56:37.768 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.9 seq 489626271747 2024-09-10T16:56:37.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:56:38.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271748 -lt 489626271747 2024-09-10T16:56:38.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:56:38.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201318 2024-09-10T16:56:38.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:56:38.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:56:38.226 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201318 2024-09-10T16:56:38.227 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:56:38.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201318 2024-09-10T16:56:38.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201318' 2024-09-10T16:56:38.228 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.10 seq 236223201318 2024-09-10T16:56:38.228 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:56:38.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201319 -lt 236223201318 2024-09-10T16:56:38.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:56:38.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:38.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:39.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-10T16:56:39.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:56:39.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:56:39.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-10T16:56:39.578 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:56:39.578 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:56:39.578 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=remap-pool 2024-09-10T16:56:40.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-10T16:56:40.610 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'remap-pool' does not exist 2024-09-10T16:56:40.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-10T16:56:41.788 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile remap-profile does not exist 2024-09-10T16:56:41.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: erasure_code_plugin_exists: local plugin=isa 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: local status 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local grepstr 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: local s 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: case `uname` in 2024-09-10T16:56:41.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: uname 2024-09-10T16:56:41.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-10T16:56:41.802 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-10T16:56:42.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: erasure_code_plugin_exists: s= 2024-09-10T16:56:42.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: local status=0 2024-09-10T16:56:42.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-10T16:56:42.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-10T16:56:43.047 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-10T16:56:43.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: erasure_code_plugin_exists: return 0 2024-09-10T16:56:43.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-10T16:56:43.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-10T16:56:43.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-10T16:56:43.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-10T16:56:43.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-10T16:56:44.649 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-isa' already exists 2024-09-10T16:56:44.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:45.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:56:45.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:45.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:56:45.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:45.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:56:45.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:56:46.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:56:46.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:56:46.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:56:46.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-10T16:56:46.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-10T16:56:46.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:56:46.714 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-10T16:56:46.714 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:10 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:56:47.193 INFO:tasks.workunit.client.0.smithi086.stderr:9' 2024-09-10T16:56:47.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-10T16:56:47.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-09-10T16:56:47.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:56:47.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-10T16:56:47.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-10T16:56:47.705 INFO:tasks.workunit.client.0.smithi086.stderr:osd.9 is already out. 2024-09-10T16:56:47.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-10T16:56:52.719 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:56:53.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:10 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-10T16:56:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:56:53.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:56:53.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-10T16:56:53.705 INFO:tasks.workunit.client.0.smithi086.stderr:osd.9 is already in. 2024-09-10T16:56:53.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:56:53.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-10T16:56:53.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-isa 2024-09-10T16:56:53.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-10T16:56:55.205 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-isa' does not exist 2024-09-10T16:56:55.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:56:55.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:55.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:56:55.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:56:55.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:56:55.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:56:56.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:56:56.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:56:56.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:56:56.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-10T16:56:56.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T16:56:56.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:56:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T16:56:56.501 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:56:57.058 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:9' 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 9 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-10T16:56:57.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-10T16:56:57.550 INFO:tasks.workunit.client.0.smithi086.stderr:osd.9 is already out. 2024-09-10T16:56:57.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:57:02.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-09-10T16:57:02.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-10T16:57:02.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:02.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-09-10T16:57:02.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-10T16:57:02.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2024-09-10T16:57:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:57:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:5' 2024-09-10T16:57:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 5 2024-09-10T16:57:03.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:03.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:03.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-09-10T16:57:03.597 INFO:tasks.workunit.client.0.smithi086.stderr:osd.9 is already in. 2024-09-10T16:57:03.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:57:03.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-10T16:57:03.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-10T16:57:03.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-10T16:57:05.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-10T16:57:05.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-10T16:57:06.745 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T16:57:06.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:57:07.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-10T16:57:07.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:57:07.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:57:09.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:10.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:10.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:57:10.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:10.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:10.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:10.348 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:10.348 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:10.904 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:57:10.904 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:10.904 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-10T16:57:10.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-10T16:57:11.748 INFO:tasks.workunit.client.0.smithi086.stderr:osd.10 is already out. 2024-09-10T16:57:11.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:57:16.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:16.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:16.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:16.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-09-10T16:57:16.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:16.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:17.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:57:17.185 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 6 2024-09-10T16:57:17.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:17.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:17.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-09-10T16:57:17.748 INFO:tasks.workunit.client.0.smithi086.stderr:osd.10 is already in. 2024-09-10T16:57:17.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-10T16:57:17.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-10T16:57:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-10T16:57:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-10T16:57:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-10T16:57:17.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:57:20.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:20.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:20.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-10T16:57:20.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-09-10T16:57:20.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:57:20.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:57:20.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:57:20.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:57:20.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:57:20.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:57:20.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:57:20.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:57:20.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:57:20.314 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:57:20.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:20.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:57:20.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:20.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:57:21.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836593 2024-09-10T16:57:21.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836593 2024-09-10T16:57:21.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593' 2024-09-10T16:57:21.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.208 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:57:21.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673065 2024-09-10T16:57:21.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673065 2024-09-10T16:57:21.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065' 2024-09-10T16:57:21.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.329 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:57:21.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509540 2024-09-10T16:57:21.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509540 2024-09-10T16:57:21.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540' 2024-09-10T16:57:21.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:57:21.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351007 2024-09-10T16:57:21.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351007 2024-09-10T16:57:21.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007' 2024-09-10T16:57:21.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:57:21.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182489 2024-09-10T16:57:21.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182489 2024-09-10T16:57:21.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489' 2024-09-10T16:57:21.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.758 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:57:21.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122074 2024-09-10T16:57:21.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122074 2024-09-10T16:57:21.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074' 2024-09-10T16:57:21.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:57:21.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533396 2024-09-10T16:57:21.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533396 2024-09-10T16:57:21.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074 6-455266533396' 2024-09-10T16:57:21.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:21.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:57:22.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691905 2024-09-10T16:57:22.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691905 2024-09-10T16:57:22.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074 6-455266533396 7-171798691905' 2024-09-10T16:57:22.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:22.113 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:57:22.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528380 2024-09-10T16:57:22.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528380 2024-09-10T16:57:22.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074 6-455266533396 7-171798691905 8-193273528380' 2024-09-10T16:57:22.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:22.239 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:57:22.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271759 2024-09-10T16:57:22.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271759 2024-09-10T16:57:22.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074 6-455266533396 7-171798691905 8-193273528380 9-489626271759' 2024-09-10T16:57:22.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:22.350 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:57:22.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201330 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201330 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673065 2-64424509540 3-347892351007 4-107374182489 5-377957122074 6-455266533396 7-171798691905 8-193273528380 9-489626271759 10-236223201330' 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836593 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:22.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:57:22.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836593 2024-09-10T16:57:22.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:22.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836593 2024-09-10T16:57:22.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-09-10T16:57:22.465 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836593 2024-09-10T16:57:22.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:57:22.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836593 -lt 21474836593 2024-09-10T16:57:22.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:22.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673065 2024-09-10T16:57:22.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:22.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:57:22.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673065 2024-09-10T16:57:22.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:22.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673065 2024-09-10T16:57:22.846 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949673065 2024-09-10T16:57:22.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-09-10T16:57:22.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:57:23.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673065 -lt 42949673065 2024-09-10T16:57:23.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:23.230 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509540 2024-09-10T16:57:23.230 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:23.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:57:23.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509540 2024-09-10T16:57:23.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:23.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509540 2024-09-10T16:57:23.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-09-10T16:57:23.233 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509540 2024-09-10T16:57:23.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:57:23.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509540 -lt 64424509540 2024-09-10T16:57:23.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:23.627 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351007 2024-09-10T16:57:23.627 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:23.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:57:23.628 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351007 2024-09-10T16:57:23.628 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:23.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351007 2024-09-10T16:57:23.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351007' 2024-09-10T16:57:23.630 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 347892351007 2024-09-10T16:57:23.630 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:57:24.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351008 -lt 347892351007 2024-09-10T16:57:24.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:24.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182489 2024-09-10T16:57:24.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:24.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:57:24.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182489 2024-09-10T16:57:24.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:24.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182489 2024-09-10T16:57:24.081 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 107374182489 2024-09-10T16:57:24.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-09-10T16:57:24.082 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:57:24.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182489 -lt 107374182489 2024-09-10T16:57:24.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:24.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122074 2024-09-10T16:57:24.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:24.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:57:24.507 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122074 2024-09-10T16:57:24.507 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:24.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122074 2024-09-10T16:57:24.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122074' 2024-09-10T16:57:24.508 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 377957122074 2024-09-10T16:57:24.508 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:57:24.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122075 -lt 377957122074 2024-09-10T16:57:24.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:24.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533396 2024-09-10T16:57:24.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:24.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:57:24.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533396 2024-09-10T16:57:24.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:24.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533396 2024-09-10T16:57:24.926 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 455266533396 2024-09-10T16:57:24.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533396' 2024-09-10T16:57:24.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:57:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533396 -lt 455266533396 2024-09-10T16:57:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:25.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691905 2024-09-10T16:57:25.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:25.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:57:25.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691905 2024-09-10T16:57:25.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:25.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691905 2024-09-10T16:57:25.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-09-10T16:57:25.344 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 171798691905 2024-09-10T16:57:25.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:57:25.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691905 -lt 171798691905 2024-09-10T16:57:25.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:25.777 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528380 2024-09-10T16:57:25.777 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:25.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:57:25.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528380 2024-09-10T16:57:25.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:25.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528380 2024-09-10T16:57:25.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528380' 2024-09-10T16:57:25.779 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 193273528380 2024-09-10T16:57:25.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:57:26.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528380 -lt 193273528380 2024-09-10T16:57:26.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:26.184 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-489626271759 2024-09-10T16:57:26.184 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:26.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:57:26.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-489626271759 2024-09-10T16:57:26.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:26.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271759 2024-09-10T16:57:26.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 489626271759' 2024-09-10T16:57:26.187 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.9 seq 489626271759 2024-09-10T16:57:26.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:57:26.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271760 -lt 489626271759 2024-09-10T16:57:26.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:26.569 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201330 2024-09-10T16:57:26.569 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:26.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:57:26.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201330 2024-09-10T16:57:26.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:26.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201330 2024-09-10T16:57:26.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201330' 2024-09-10T16:57:26.572 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.10 seq 236223201330 2024-09-10T16:57:26.572 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:57:26.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201331 -lt 236223201330 2024-09-10T16:57:26.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:57:26.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:26.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:27.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-10T16:57:27.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:57:27.517 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:57:27.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:57:27.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:57:27.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:57:27.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:57:27.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:57:27.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-10T16:57:27.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:57:27.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:27.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:28.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:28.456 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-09-10T16:57:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:28.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-09-10T16:57:29.369 INFO:tasks.workunit.client.0.smithi086.stderr:osd.5 is already out. 2024-09-10T16:57:29.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:29.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-09-10T16:57:31.151 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 is already out. 2024-09-10T16:57:31.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-09-10T16:57:31.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:57:31.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:57:31.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:57:31.164 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:57:31.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:57:31.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:57:31.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:57:31.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:57:31.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:57:31.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:57:31.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:57:31.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:57:31.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:57:31.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:57:31.253 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:32.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:57:32.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836597 2024-09-10T16:57:32.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836597 2024-09-10T16:57:32.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597' 2024-09-10T16:57:32.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:32.808 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:57:33.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673069 2024-09-10T16:57:33.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673069 2024-09-10T16:57:33.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069' 2024-09-10T16:57:33.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:33.241 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:57:33.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509545 2024-09-10T16:57:33.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509545 2024-09-10T16:57:33.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545' 2024-09-10T16:57:33.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:33.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:57:33.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351011 2024-09-10T16:57:33.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351011 2024-09-10T16:57:33.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011' 2024-09-10T16:57:33.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:33.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:57:34.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182493 2024-09-10T16:57:34.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182493 2024-09-10T16:57:34.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493' 2024-09-10T16:57:34.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:34.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:57:34.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122079 2024-09-10T16:57:34.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122079 2024-09-10T16:57:34.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079' 2024-09-10T16:57:34.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:34.313 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:57:34.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533401 2024-09-10T16:57:34.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533401 2024-09-10T16:57:34.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079 6-455266533401' 2024-09-10T16:57:34.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:34.693 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:57:34.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691909 2024-09-10T16:57:34.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691909 2024-09-10T16:57:34.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079 6-455266533401 7-171798691909' 2024-09-10T16:57:34.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:34.805 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:57:34.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528384 2024-09-10T16:57:34.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528384 2024-09-10T16:57:34.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079 6-455266533401 7-171798691909 8-193273528384' 2024-09-10T16:57:34.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:34.917 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:57:35.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271764 2024-09-10T16:57:35.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271764 2024-09-10T16:57:35.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079 6-455266533401 7-171798691909 8-193273528384 9-489626271764' 2024-09-10T16:57:35.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:35.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:57:35.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201335 2024-09-10T16:57:35.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201335 2024-09-10T16:57:35.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673069 2-64424509545 3-347892351011 4-107374182493 5-377957122079 6-455266533401 7-171798691909 8-193273528384 9-489626271764 10-236223201335' 2024-09-10T16:57:35.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:35.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836597 2024-09-10T16:57:35.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:35.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:57:35.238 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836597 2024-09-10T16:57:35.238 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:35.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836597 2024-09-10T16:57:35.239 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836597 2024-09-10T16:57:35.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-09-10T16:57:35.240 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:57:35.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836597 -lt 21474836597 2024-09-10T16:57:35.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:35.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673069 2024-09-10T16:57:35.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:35.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:57:35.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673069 2024-09-10T16:57:35.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:35.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673069 2024-09-10T16:57:35.662 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949673069 2024-09-10T16:57:35.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-09-10T16:57:35.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:57:36.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673070 -lt 42949673069 2024-09-10T16:57:36.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:36.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509545 2024-09-10T16:57:36.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:57:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509545 2024-09-10T16:57:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509545 2024-09-10T16:57:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2024-09-10T16:57:36.081 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509545 2024-09-10T16:57:36.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:57:36.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509545 -lt 64424509545 2024-09-10T16:57:36.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:36.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351011 2024-09-10T16:57:36.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:36.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:57:36.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351011 2024-09-10T16:57:36.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:36.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351011 2024-09-10T16:57:36.507 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 347892351011 2024-09-10T16:57:36.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351011' 2024-09-10T16:57:36.507 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:57:36.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351012 -lt 347892351011 2024-09-10T16:57:36.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:36.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182493 2024-09-10T16:57:36.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:36.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:57:36.924 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182493 2024-09-10T16:57:36.924 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:36.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182493 2024-09-10T16:57:36.925 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 107374182493 2024-09-10T16:57:36.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-09-10T16:57:36.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:57:37.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182493 -lt 107374182493 2024-09-10T16:57:37.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:37.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122079 2024-09-10T16:57:37.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:37.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:57:37.334 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122079 2024-09-10T16:57:37.334 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:37.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122079 2024-09-10T16:57:37.335 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 377957122079 2024-09-10T16:57:37.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122079' 2024-09-10T16:57:37.336 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:57:37.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122079 -lt 377957122079 2024-09-10T16:57:37.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:37.720 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533401 2024-09-10T16:57:37.721 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:37.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:57:37.722 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533401 2024-09-10T16:57:37.722 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:37.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533401 2024-09-10T16:57:37.723 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 455266533401 2024-09-10T16:57:37.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533401' 2024-09-10T16:57:37.724 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:57:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533401 -lt 455266533401 2024-09-10T16:57:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:38.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691909 2024-09-10T16:57:38.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:57:38.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691909 2024-09-10T16:57:38.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:38.120 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 171798691909 2024-09-10T16:57:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691909 2024-09-10T16:57:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691909' 2024-09-10T16:57:38.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:57:38.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691909 2024-09-10T16:57:38.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:38.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528384 2024-09-10T16:57:38.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:38.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:57:38.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528384 2024-09-10T16:57:38.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:38.566 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 193273528384 2024-09-10T16:57:38.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528384 2024-09-10T16:57:38.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528384' 2024-09-10T16:57:38.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:57:38.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528385 -lt 193273528384 2024-09-10T16:57:38.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:38.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-489626271764 2024-09-10T16:57:38.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:38.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:57:38.992 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-489626271764 2024-09-10T16:57:38.992 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:38.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271764 2024-09-10T16:57:38.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 489626271764' 2024-09-10T16:57:38.994 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.9 seq 489626271764 2024-09-10T16:57:38.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:57:39.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271764 -lt 489626271764 2024-09-10T16:57:39.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:39.385 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201335 2024-09-10T16:57:39.385 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:39.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:57:39.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201335 2024-09-10T16:57:39.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:39.387 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.10 seq 236223201335 2024-09-10T16:57:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201335 2024-09-10T16:57:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201335' 2024-09-10T16:57:39.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:57:39.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201335 -lt 236223201335 2024-09-10T16:57:39.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:57:39.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:39.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:40.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-10T16:57:40.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:57:40.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:57:40.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-10T16:57:40.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:57:40.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:40.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:41.272 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:41.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T16:57:41.736 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-09-10T16:57:41.738 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:41.738 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2024-09-10T16:57:42.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:42.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:42.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:42.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-09-10T16:57:42.728 INFO:tasks.workunit.client.0.smithi086.stderr:osd.5 is already in. 2024-09-10T16:57:42.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-09-10T16:57:42.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-09-10T16:57:44.306 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 is already in. 2024-09-10T16:57:44.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-09-10T16:57:44.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T16:57:44.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T16:57:44.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T16:57:44.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T16:57:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T16:57:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T16:57:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T16:57:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T16:57:44.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T16:57:44.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:8 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T16:57:45.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:45.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T16:57:45.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836601 2024-09-10T16:57:45.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836601 2024-09-10T16:57:45.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601' 2024-09-10T16:57:45.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:45.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T16:57:45.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673074 2024-09-10T16:57:45.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673074 2024-09-10T16:57:45.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074' 2024-09-10T16:57:45.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:45.578 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T16:57:45.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509549 2024-09-10T16:57:45.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509549 2024-09-10T16:57:45.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549' 2024-09-10T16:57:45.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:45.699 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T16:57:45.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892351016 2024-09-10T16:57:45.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892351016 2024-09-10T16:57:45.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016' 2024-09-10T16:57:45.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:45.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T16:57:46.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182498 2024-09-10T16:57:46.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182498 2024-09-10T16:57:46.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498' 2024-09-10T16:57:46.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T16:57:46.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122083 2024-09-10T16:57:46.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122083 2024-09-10T16:57:46.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083' 2024-09-10T16:57:46.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T16:57:46.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533405 2024-09-10T16:57:46.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533405 2024-09-10T16:57:46.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083 6-455266533405' 2024-09-10T16:57:46.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T16:57:46.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691913 2024-09-10T16:57:46.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691913 2024-09-10T16:57:46.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083 6-455266533405 7-171798691913' 2024-09-10T16:57:46.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T16:57:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528388 2024-09-10T16:57:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528388 2024-09-10T16:57:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083 6-455266533405 7-171798691913 8-193273528388' 2024-09-10T16:57:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-10T16:57:46.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271768 2024-09-10T16:57:46.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271768 2024-09-10T16:57:46.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083 6-455266533405 7-171798691913 8-193273528388 9-489626271768' 2024-09-10T16:57:46.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T16:57:46.959 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-10T16:57:47.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=236223201339 2024-09-10T16:57:47.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 236223201339 2024-09-10T16:57:47.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836601 1-42949673074 2-64424509549 3-347892351016 4-107374182498 5-377957122083 6-455266533405 7-171798691913 8-193273528388 9-489626271768 10-236223201339' 2024-09-10T16:57:47.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:47.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836601 2024-09-10T16:57:47.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:47.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T16:57:47.072 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836601 2024-09-10T16:57:47.072 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:47.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836601 2024-09-10T16:57:47.073 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 21474836601 2024-09-10T16:57:47.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2024-09-10T16:57:47.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T16:57:47.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836601 -lt 21474836601 2024-09-10T16:57:47.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:47.457 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673074 2024-09-10T16:57:47.457 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:47.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T16:57:47.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673074 2024-09-10T16:57:47.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:47.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673074 2024-09-10T16:57:47.460 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 42949673074 2024-09-10T16:57:47.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-09-10T16:57:47.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T16:57:47.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673074 -lt 42949673074 2024-09-10T16:57:47.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:47.918 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509549 2024-09-10T16:57:47.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:47.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T16:57:47.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509549 2024-09-10T16:57:47.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:47.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509549 2024-09-10T16:57:47.921 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 64424509549 2024-09-10T16:57:47.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-09-10T16:57:47.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T16:57:48.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509549 -lt 64424509549 2024-09-10T16:57:48.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:48.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-347892351016 2024-09-10T16:57:48.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:48.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T16:57:48.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-347892351016 2024-09-10T16:57:48.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:48.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892351016 2024-09-10T16:57:48.383 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 347892351016 2024-09-10T16:57:48.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 347892351016' 2024-09-10T16:57:48.384 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T16:57:48.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892351016 -lt 347892351016 2024-09-10T16:57:48.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:48.798 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182498 2024-09-10T16:57:48.798 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:48.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T16:57:48.800 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182498 2024-09-10T16:57:48.800 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:48.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182498 2024-09-10T16:57:48.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182498' 2024-09-10T16:57:48.801 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 107374182498 2024-09-10T16:57:48.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T16:57:49.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182498 -lt 107374182498 2024-09-10T16:57:49.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:49.217 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-377957122083 2024-09-10T16:57:49.217 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:49.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T16:57:49.218 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-377957122083 2024-09-10T16:57:49.218 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:49.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122083 2024-09-10T16:57:49.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 377957122083' 2024-09-10T16:57:49.220 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 377957122083 2024-09-10T16:57:49.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T16:57:49.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122084 -lt 377957122083 2024-09-10T16:57:49.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:49.643 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-455266533405 2024-09-10T16:57:49.643 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:49.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T16:57:49.645 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-455266533405 2024-09-10T16:57:49.645 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:49.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533405 2024-09-10T16:57:49.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 455266533405' 2024-09-10T16:57:49.646 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 455266533405 2024-09-10T16:57:49.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T16:57:50.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533406 -lt 455266533405 2024-09-10T16:57:50.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:50.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691913 2024-09-10T16:57:50.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:50.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T16:57:50.048 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691913 2024-09-10T16:57:50.049 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:50.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691913 2024-09-10T16:57:50.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691913' 2024-09-10T16:57:50.050 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 171798691913 2024-09-10T16:57:50.050 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T16:57:50.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691914 -lt 171798691913 2024-09-10T16:57:50.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:50.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528388 2024-09-10T16:57:50.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:50.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T16:57:50.433 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528388 2024-09-10T16:57:50.433 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:50.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528388 2024-09-10T16:57:50.434 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 193273528388 2024-09-10T16:57:50.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528388' 2024-09-10T16:57:50.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T16:57:50.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528389 -lt 193273528388 2024-09-10T16:57:50.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:50.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-489626271768 2024-09-10T16:57:50.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:50.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-10T16:57:50.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-489626271768 2024-09-10T16:57:50.818 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:50.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271768 2024-09-10T16:57:50.819 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.9 seq 489626271768 2024-09-10T16:57:50.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 489626271768' 2024-09-10T16:57:50.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-10T16:57:51.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271769 -lt 489626271768 2024-09-10T16:57:51.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T16:57:51.249 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 10-236223201339 2024-09-10T16:57:51.249 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T16:57:51.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=10 2024-09-10T16:57:51.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 10-236223201339 2024-09-10T16:57:51.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T16:57:51.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=236223201339 2024-09-10T16:57:51.251 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.10 seq 236223201339 2024-09-10T16:57:51.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.10 seq 236223201339' 2024-09-10T16:57:51.252 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-10T16:57:51.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 236223201340 -lt 236223201339 2024-09-10T16:57:51.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T16:57:51.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:51.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:52.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 28 == 0 2024-09-10T16:57:52.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T16:57:52.230 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T16:57:52.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T16:57:52.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T16:57:52.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T16:57:52.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T16:57:52.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T16:57:52.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=28 2024-09-10T16:57:52.648 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T16:57:52.648 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T16:57:52.648 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T16:57:53.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 = 28 2024-09-10T16:57:53.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T16:57:53.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T16:57:53.164 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-10T16:57:53.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T16:57:53.165 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:53.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-10T16:57:53.165 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:10' 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-10T16:57:53.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:57:53.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-10T16:57:53.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-jerasure 2024-09-10T16:57:53.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T16:57:54.113 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T16:57:54.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-10T16:57:54.777 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-10T16:57:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-10T16:57:55.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-10T16:57:55.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-10T16:57:56.840 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-lrc-a' already exists 2024-09-10T16:57:56.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:57:57.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-10T16:57:57.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:57:57.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-10T16:57:57.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:57.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:57:57.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:57:57.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:57:57.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:57:58.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:57:59.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:57:59.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:57:59.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-10T16:57:59.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-10T16:57:59.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:57:59.183 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-10T16:57:59.183 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 3 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-09-10T16:57:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:57:59.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-10T16:57:59.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-10T16:58:01.088 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 is already out. 2024-09-10T16:58:01.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:58:06.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-10T16:58:06.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2024-09-10T16:58:06.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:58:06.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-09-10T16:58:06.105 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-10T16:58:06.105 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:58:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2024-09-10T16:58:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:58:06.499 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T16:58:06.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 7 2024-09-10T16:58:06.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:58:06.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:58:06.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-09-10T16:58:07.039 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 is already in. 2024-09-10T16:58:07.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:58:07.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-10T16:58:07.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc-a 2024-09-10T16:58:07.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-10T16:58:08.637 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-lrc-a' does not exist 2024-09-10T16:58:08.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-10T16:58:09.586 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-10T16:58:09.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:58:09.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-10T16:58:09.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-10T16:58:09.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-10T16:58:09.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-10T16:58:09.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-10T16:58:10.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-10T16:58:10.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-10T16:58:11.775 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-lrc' already exists 2024-09-10T16:58:11.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:58:12.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:12.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:58:12.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:12.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:58:12.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:58:13.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:58:13.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:58:13.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:58:13.641 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-10T16:58:13.642 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-10T16:58:13.642 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:58:13.642 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-10T16:58:13.643 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:58:14.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-10T16:58:14.081 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:10 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-10T16:58:14.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-10T16:58:15.311 INFO:tasks.workunit.client.0.smithi086.stderr:osd.7 is already out. 2024-09-10T16:58:15.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-10T16:58:20.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:10 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T16:58:20.724 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:58:20.725 INFO:tasks.workunit.client.0.smithi086.stderr:4' 2024-09-10T16:58:20.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-10T16:58:20.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:58:20.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:58:20.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-09-10T16:58:21.287 INFO:tasks.workunit.client.0.smithi086.stderr:osd.7 is already in. 2024-09-10T16:58:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:58:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-10T16:58:21.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc 2024-09-10T16:58:21.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-10T16:58:22.767 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-lrc' does not exist 2024-09-10T16:58:22.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-10T16:58:24.222 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile profile-lrc does not exist 2024-09-10T16:58:24.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-10T16:58:24.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-10T16:58:24.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-10T16:58:24.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-10T16:58:24.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-10T16:58:24.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-10T16:58:25.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-10T16:58:25.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-10T16:58:26.102 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-shec' already exists 2024-09-10T16:58:26.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:58:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-10T16:58:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-10T16:58:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-10T16:58:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-10T16:58:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-10T16:58:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:58:29.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:58:29.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-10T16:58:29.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-10T16:58:29.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-10T16:58:29.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:58:29.269 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-10T16:58:29.269 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:58:29.725 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-10T16:58:29.725 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:58:29.725 INFO:tasks.workunit.client.0.smithi086.stderr:1' 2024-09-10T16:58:29.725 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 1 2024-09-10T16:58:29.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-09-10T16:58:29.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-10T16:58:29.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-10T16:58:29.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-10T16:58:30.533 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 is already out. 2024-09-10T16:58:30.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-09-10T16:58:35.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-09-10T16:58:35.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2024-09-10T16:58:35.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-10T16:58:35.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-09-10T16:58:35.547 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-10T16:58:35.547 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T16:58:36.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2024-09-10T16:58:36.007 INFO:tasks.workunit.client.0.smithi086.stderr:9 2024-09-10T16:58:36.007 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T16:58:36.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 6 2024-09-10T16:58:36.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-10T16:58:36.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-10T16:58:36.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-09-10T16:58:36.706 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 is already in. 2024-09-10T16:58:36.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-10T16:58:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-10T16:58:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-shec 2024-09-10T16:58:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-10T16:58:38.396 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-shec' does not exist 2024-09-10T16:58:38.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-10T16:58:39.650 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile profile-shec does not exist 2024-09-10T16:58:39.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-10T16:58:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=ecpool 2024-09-10T16:58:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-10T16:58:40.592 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'ecpool' does not exist 2024-09-10T16:58:40.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-10T16:58:40.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-10T16:58:40.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:58:40.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-10T16:58:40.604 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:58:40.604 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:58:40.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:58:40.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:58:40.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:58:40.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:58:40.718 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:58:40.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:58:40.719 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:58:40.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:58:40.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:58:40.721 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:58:40.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:58:40.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:58:40.722 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:58:40.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:58:40.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-10T16:58:40.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:58:40.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:40.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:58:40.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55334 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-code 0 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T16:58:40.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:58:40.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:58:40.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:58:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:58:40.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:58:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:58:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:58:40.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:58:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:58:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:58:40.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:58:40.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T16:58:40.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2024-09-10T16:58:40.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:58:40.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:40.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55334 2024-09-10T16:58:40.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55334 2024-09-10T16:58:40.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:58:40.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:58:40.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T16:58:40.833 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T16:58:40.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T16:58:40.898 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-10T16:58:40.898 DEBUG:teuthology.orchestra.run.smithi086:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-10T16:58:40.960 INFO:tasks.workunit.client.0.smithi086.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/test-erasure-eio 2024-09-10T16:58:40.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none ' 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-10T16:58:40.968 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_object_attr_read_error 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_recovery_multiple_errors 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_recovery_multiple_objects 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_recovery_unfound 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_ec_single_recovery_error 2024-09-10T16:58:40.970 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_bad_size_shard_0 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_bad_size_shard_1 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_subread_missing 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T16:58:40.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T16:58:40.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T16:58:40.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T16:58:40.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T16:58:40.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T16:58:40.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T16:58:40.973 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T16:58:40.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T16:58:40.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T16:58:40.974 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T16:58:40.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:58:40.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T16:58:40.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T16:58:40.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T16:58:40.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:40.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:40.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T16:58:40.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T16:58:40.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T16:58:40.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T16:58:40.979 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T16:58:40.979 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:40.979 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:40.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T16:58:40.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T16:58:40.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T16:58:40.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:58:41.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T16:58:41.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T16:58:41.666 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:58:41.667 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T16:58:41.746 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T16:58:41.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T16:58:41.748 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T16:58:41.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T16:58:41.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T16:58:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T16:58:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T16:58:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T16:58:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T16:58:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:58:42.669 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:58:42.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T16:58:42.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T16:58:42.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T16:58:42.989 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T16:58:42.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T16:58:44.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T16:58:44.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T16:58:44.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T16:58:44.071 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T16:58:44.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T16:58:44.079 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:58:41.504+0000 7f015480bc80 0 load: jerasure load: lrc 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: local objname=myobject 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: local lastobj=300 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:535: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: setup_osds 5 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T16:58:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-10T16:58:44.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-10T16:58:44.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T16:58:44.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T16:58:44.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:58:44.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=29f0f57f-805a-4303-a072-942df019311b 2024-09-10T16:58:44.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 29f0f57f-805a-4303-a072-942df019311b' 2024-09-10T16:58:44.089 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 29f0f57f-805a-4303-a072-942df019311b 2024-09-10T16:58:44.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:58:44.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEeuBmyGtHBhAAuSvZa60MR88+v+YNzoWIBA== 2024-09-10T16:58:44.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEeuBmyGtHBhAAuSvZa60MR88+v+YNzoWIBA=="}' 2024-09-10T16:58:44.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 29f0f57f-805a-4303-a072-942df019311b -i td/test-erasure-eio/0/new.json 2024-09-10T16:58:44.365 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:58:44.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T16:58:44.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEeuBmyGtHBhAAuSvZa60MR88+v+YNzoWIBA== --osd-uuid 29f0f57f-805a-4303-a072-942df019311b 2024-09-10T16:58:44.409 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:44.407+0000 7fa57d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:44.434 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:44.433+0000 7fa57d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:44.435 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:44.434+0000 7fa57d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:44.435 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:44.434+0000 7fa57d413640 -1 bdev(0x55731f86bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T16:58:44.436 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:44.434+0000 7fa57d413640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T16:58:50.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T16:58:50.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:58:50.791 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T16:58:50.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T16:58:50.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:58:51.177 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T16:58:51.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T16:58:51.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:58:51.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:58:51.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:58:51.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:58:51.208 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:51.207+0000 7fb357c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:51.214 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:51.212+0000 7fb357c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:51.215 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:51.214+0000 7fb357c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:58:51.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T16:58:51.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:58:51.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T16:58:51.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:58:51.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:58:51.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:51.586 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:58:51.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:58:51.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:51.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:52.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:58:52.561 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:52.559+0000 7fb357c17640 -1 Falling back to public interface 2024-09-10T16:58:53.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:58:53.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:53.013 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:58:53.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:58:53.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:53.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:53.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:58:54.167 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:58:54.165+0000 7fb357c17640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:58:54.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:58:54.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:54.449 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:58:54.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:58:54.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:54.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:54.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:58:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:58:55.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:58:55.904 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:58:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:55.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:56.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:58:57.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:58:57.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:57.317 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:58:57.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:58:57.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:57.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:57.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:58:58.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:58:58.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:58:58.716 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:58:58.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:58:58.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:58:58.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:58:59.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:00.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:00.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:00.200 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:59:00.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:59:00.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:00.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:00.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:01.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:01.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:01.600 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:59:01.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:59:01.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:01.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:02.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:03.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:03.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:03.022 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:59:03.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:59:03.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:03.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:03.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:04.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:04.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:04.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:59:04.449 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:59:04.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:04.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:04.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:05.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:05.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:05.910 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:59:05.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:59:05.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:05.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:07.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:08.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:08.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:08.022 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:59:08.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:59:08.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:08.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:08.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:09.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:09.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:09.741 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:59:09.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:59:09.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:09.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:10.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:11.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:11.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:11.544 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:59:11.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:59:11.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:11.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:12.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:13.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:13.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:13.346 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:59:13.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:59:13.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:13.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:14.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:15.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:15.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:15.208 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:59:15.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:59:15.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:15.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:15.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:16.993 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:59:16.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:59:16.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:16.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T16:59:17.457 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1454624556,v1:127.0.0.1:6803/1454624556] [v2:127.0.0.1:6804/1454624556,v1:127.0.0.1:6805/1454624556] exists,up 29f0f57f-805a-4303-a072-942df019311b 2024-09-10T16:59:17.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T16:59:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:59:17.463 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:59:17.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78a7870a-0071-4be0-8318-f47de04971c7 2024-09-10T16:59:17.465 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 78a7870a-0071-4be0-8318-f47de04971c7 2024-09-10T16:59:17.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 78a7870a-0071-4be0-8318-f47de04971c7' 2024-09-10T16:59:17.466 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:59:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDleuBmTNScHBAAuYZ2Qy+2D1ZT9jEXX5iYeA== 2024-09-10T16:59:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDleuBmTNScHBAAuYZ2Qy+2D1ZT9jEXX5iYeA=="}' 2024-09-10T16:59:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78a7870a-0071-4be0-8318-f47de04971c7 -i td/test-erasure-eio/1/new.json 2024-09-10T16:59:18.150 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:59:18.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T16:59:18.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDleuBmTNScHBAAuYZ2Qy+2D1ZT9jEXX5iYeA== --osd-uuid 78a7870a-0071-4be0-8318-f47de04971c7 2024-09-10T16:59:18.195 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:18.194+0000 7ff252aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:18.197 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:18.196+0000 7ff252aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:18.198 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:18.197+0000 7ff252aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:18.198 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:18.197+0000 7ff252aa5640 -1 bdev(0x55adeda43c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T16:59:18.198 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:18.197+0000 7ff252aa5640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T16:59:24.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T16:59:24.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:59:24.171 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T16:59:24.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T16:59:24.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:59:24.717 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T16:59:24.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T16:59:24.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:59:24.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:59:24.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:59:24.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:59:24.749 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:24.747+0000 7f288ac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:24.754 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:24.752+0000 7f288ac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:24.755 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:24.754+0000 7f288ac9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:25.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:25.168 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:59:25.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:59:25.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:25.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:25.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:25.602 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:25.601+0000 7f288ac9c640 -1 Falling back to public interface 2024-09-10T16:59:26.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:26.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:26.550 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T16:59:26.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T16:59:26.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:26.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:26.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:27.398 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:27.396+0000 7f288ac9c640 -1 osd.1 0 log_to_monitors true 2024-09-10T16:59:27.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:27.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:27.964 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:59:27.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T16:59:27.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:27.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:28.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:29.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:29.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:29.404 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T16:59:29.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T16:59:29.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:29.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:29.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:30.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:30.844 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T16:59:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T16:59:30.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:30.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:31.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:32.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:32.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:32.501 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T16:59:32.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T16:59:32.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:32.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:32.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:33.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:33.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:33.900 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T16:59:33.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T16:59:33.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:33.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:34.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:35.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:35.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:35.330 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T16:59:35.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T16:59:35.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:35.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:35.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:36.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:36.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:36.746 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T16:59:36.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T16:59:36.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:36.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:37.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:38.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:38.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:38.204 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T16:59:38.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T16:59:38.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:38.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:38.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:39.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:39.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:39.634 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T16:59:39.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T16:59:39.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:39.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:40.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:41.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:41.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:41.334 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T16:59:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T16:59:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:42.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:43.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:43.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:43.540 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T16:59:43.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T16:59:43.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:43.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:44.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:45.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:45.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:45.433 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T16:59:45.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T16:59:45.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:45.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:46.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:47.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:47.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:47.603 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T16:59:47.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T16:59:47.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:47.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:48.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:49.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:49.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:49.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T16:59:49.390 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T16:59:49.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:49.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:49.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:50.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T16:59:50.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:50.942 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T16:59:50.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T16:59:50.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:50.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T16:59:51.432 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3165762611,v1:127.0.0.1:6811/3165762611] [v2:127.0.0.1:6812/3165762611,v1:127.0.0.1:6813/3165762611] exists,up 78a7870a-0071-4be0-8318-f47de04971c7 2024-09-10T16:59:51.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T16:59:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T16:59:51.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T16:59:51.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T16:59:51.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 43a9a5a1-4741-41af-8338-5d68efcbb7df' 2024-09-10T16:59:51.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T16:59:51.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAHe+Bm2o8xGxAAXqlRN/X1GLU32AQMeiNXGA== 2024-09-10T16:59:51.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAHe+Bm2o8xGxAAXqlRN/X1GLU32AQMeiNXGA=="}' 2024-09-10T16:59:51.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43a9a5a1-4741-41af-8338-5d68efcbb7df -i td/test-erasure-eio/2/new.json 2024-09-10T16:59:51.905 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T16:59:51.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T16:59:51.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHe+Bm2o8xGxAAXqlRN/X1GLU32AQMeiNXGA== --osd-uuid 43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T16:59:51.949 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:51.948+0000 7f524000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:52.048 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:52.047+0000 7f524000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:52.049 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:52.048+0000 7f524000e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:52.049 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:52.048+0000 7f524000e640 -1 bdev(0x55ee58c6fc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T16:59:52.049 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:52.048+0000 7f524000e640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T16:59:58.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T16:59:58.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T16:59:58.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T16:59:58.084 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T16:59:58.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T16:59:58.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T16:59:58.622 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T16:59:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T16:59:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T16:59:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T16:59:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T16:59:58.654 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:58.652+0000 7f009791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:58.666 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:58.665+0000 7f009791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:58.667 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:58.666+0000 7f009791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T16:59:59.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T16:59:59.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T16:59:59.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T16:59:59.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T16:59:59.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T16:59:59.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T16:59:59.091 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T16:59:59.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T16:59:59.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T16:59:59.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T16:59:59.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T16:59:59.783 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T16:59:59.782+0000 7f009791b640 -1 Falling back to public interface 2024-09-10T17:00:00.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:00.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:00.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:00:00.526 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:00:00.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:00.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:01.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:01.620 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:01.618+0000 7f009791b640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:00:02.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:02.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:02.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:00:02.011 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:00:02.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:02.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:02.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:03.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:03.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:03.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:00:03.457 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:00:03.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:03.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:03.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:04.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:04.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:04.972 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:00:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:00:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:05.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:06.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:06.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:06.462 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:00:06.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:00:06.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:06.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:06.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:07.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:07.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:07.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:00:07.861 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:00:07.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:07.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:08.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:09.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:09.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:09.280 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:00:09.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:00:09.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:09.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:09.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:10.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:10.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:10.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:00:10.787 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:00:10.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:10.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:11.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:12.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:12.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:12.243 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:00:12.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:00:12.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:12.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:12.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:13.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:13.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:13.765 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:00:13.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:00:13.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:13.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:14.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:15.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:15.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:15.204 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:00:15.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:00:15.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:15.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:16.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:17.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:17.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:17.164 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:00:17.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:00:17.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:17.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:17.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:18.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:18.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:18.826 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:00:18.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:00:18.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:18.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:19.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:20.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:20.707 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:00:20.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:20.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:00:20.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:20.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:21.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:22.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:22.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:22.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:00:22.313 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:00:22.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:22.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:23.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:24.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:24.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:24.018 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:00:24.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:00:24.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:24.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:24.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:25.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:25.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:25.744 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:00:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:00:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:25.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:00:26.435 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/793831892,v1:127.0.0.1:6819/793831892] [v2:127.0.0.1:6820/793831892,v1:127.0.0.1:6821/793831892] exists,up 43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:00:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:00:26.445 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:00:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3744515e-d51a-486f-84da-984af42d62fc 2024-09-10T17:00:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 3744515e-d51a-486f-84da-984af42d62fc' 2024-09-10T17:00:26.448 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 3744515e-d51a-486f-84da-984af42d62fc 2024-09-10T17:00:26.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:00:26.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqe+BminJlGxAATE4sd+LGuWTJzJOU5hjr+w== 2024-09-10T17:00:26.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqe+BminJlGxAATE4sd+LGuWTJzJOU5hjr+w=="}' 2024-09-10T17:00:26.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3744515e-d51a-486f-84da-984af42d62fc -i td/test-erasure-eio/3/new.json 2024-09-10T17:00:27.178 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:00:27.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:00:27.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqe+BminJlGxAATE4sd+LGuWTJzJOU5hjr+w== --osd-uuid 3744515e-d51a-486f-84da-984af42d62fc 2024-09-10T17:00:27.224 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:27.222+0000 7f7f8fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:27.245 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:27.244+0000 7f7f8fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:27.246 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:27.245+0000 7f7f8fb45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:27.246 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:27.245+0000 7f7f8fb45640 -1 bdev(0x561ce61c5c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:00:27.247 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:27.245+0000 7f7f8fb45640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:00:32.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:00:32.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:00:32.909 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:00:32.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:00:32.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:00:33.449 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:00:33.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:00:33.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:00:33.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:00:33.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:00:33.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:00:33.480 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:33.479+0000 7f314a334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:33.500 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:33.499+0000 7f314a334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:33.501 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:33.500+0000 7f314a334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:00:33.873 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:00:33.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:33.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:00:33.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:33.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:34.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:34.582 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:34.581+0000 7f314a334640 -1 Falling back to public interface 2024-09-10T17:00:35.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:35.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:35.300 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:00:35.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:00:35.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:35.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:35.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:36.413 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:00:36.411+0000 7f314a334640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:00:36.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:36.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:36.691 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:00:36.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:00:36.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:36.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:37.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:38.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:38.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:38.284 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:00:38.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:00:38.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:38.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:38.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:39.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:39.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:39.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:00:39.961 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:00:39.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:39.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:40.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:00:41.454 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:00:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:41.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:42.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:42.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:42.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:00:42.852 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:00:42.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:42.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:43.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:44.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:44.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:44.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:00:44.488 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:00:44.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:44.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:44.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:45.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:45.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:45.895 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:00:45.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:00:45.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:45.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:46.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:47.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:47.394 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:00:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:00:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:47.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:47.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:48.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:48.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:48.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:00:48.814 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:00:48.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:48.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:49.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:50.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:50.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:50.429 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:00:50.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:00:50.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:50.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:51.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:52.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:52.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:52.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:00:52.084 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:00:52.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:52.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:52.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:53.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:53.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:53.627 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:00:53.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:00:53.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:53.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:54.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:55.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:55.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:55.200 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:00:55.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:00:55.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:55.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:55.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:56.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:56.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:56.799 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:00:56.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:00:56.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:56.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:57.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:00:58.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:00:58.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:00:58.755 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:00:58.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:00:58.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:00:58.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:00:59.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:00.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:00.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:00.507 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:01:00.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:01:00.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:00.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:01:00.946 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2878844701,v1:127.0.0.1:6827/2878844701] [v2:127.0.0.1:6828/2878844701,v1:127.0.0.1:6829/2878844701] exists,up 3744515e-d51a-486f-84da-984af42d62fc 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:01:00.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:01:00.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:01:00.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f0bb1a41-228d-4747-b8a8-02566db8767a 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 f0bb1a41-228d-4747-b8a8-02566db8767a' 2024-09-10T17:01:00.954 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 f0bb1a41-228d-4747-b8a8-02566db8767a 2024-09-10T17:01:00.955 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:01:00.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMe+Bm8z3JORAAyoGWBB81ltjCtu4R1Lj1lQ== 2024-09-10T17:01:00.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMe+Bm8z3JORAAyoGWBB81ltjCtu4R1Lj1lQ=="}' 2024-09-10T17:01:00.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f0bb1a41-228d-4747-b8a8-02566db8767a -i td/test-erasure-eio/4/new.json 2024-09-10T17:01:01.585 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:01:01.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:01:01.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMe+Bm8z3JORAAyoGWBB81ltjCtu4R1Lj1lQ== --osd-uuid f0bb1a41-228d-4747-b8a8-02566db8767a 2024-09-10T17:01:01.631 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:01.629+0000 7f6454751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:01.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:01.631+0000 7f6454751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:01.633 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:01.632+0000 7f6454751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:01.633 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:01.632+0000 7f6454751640 -1 bdev(0x556b30b0fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:01:01.634 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:01.632+0000 7f6454751640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:01:08.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:01:08.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:01:08.838 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:01:08.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:01:08.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:01:09.376 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:01:09.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:01:09.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:01:09.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:01:09.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:01:09.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:01:09.407 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:09.406+0000 7fc3e1934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:09.417 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:09.415+0000 7fc3e1934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:09.418 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:09.416+0000 7fc3e1934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:01:09.840 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:01:09.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:09.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:10.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:10.283 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:10.281+0000 7fc3e1934640 -1 Falling back to public interface 2024-09-10T17:01:11.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:11.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:11.240 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:01:11.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:01:11.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:11.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:11.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:12.054 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:01:12.053+0000 7fc3e1934640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:01:12.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:12.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:12.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:01:12.632 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:01:12.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:12.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:13.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:14.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:14.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:14.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:01:14.161 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:01:14.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:14.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:14.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:15.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:15.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:15.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:01:15.837 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:01:15.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:15.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:16.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:17.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:17.244 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:01:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:01:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:17.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:01:18.641 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:01:18.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:18.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:19.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:20.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:20.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:20.258 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:01:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:01:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:20.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:21.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:21.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:21.667 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:01:21.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:01:21.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:21.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:23.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:23.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:23.191 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:01:23.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:01:23.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:23.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:23.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:24.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:24.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:24.668 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:01:24.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:01:24.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:24.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:25.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:26.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:26.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:26.398 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:01:26.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:01:26.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:26.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:27.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:28.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:28.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:28.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:01:28.066 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:01:28.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:28.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:28.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:29.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:29.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:29.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:01:29.868 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:01:29.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:29.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:30.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:31.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:31.626 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:01:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:01:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:32.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:33.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:33.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:33.427 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:01:33.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:01:33.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:33.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:34.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:35.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:35.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:35.094 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:01:35.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:01:35.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:35.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:35.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:01:36.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:01:36.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:01:36.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:01:36.609 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:01:36.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:01:36.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:01:37.054 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2588582737,v1:127.0.0.1:6835/2588582737] [v2:127.0.0.1:6836/2588582737,v1:127.0.0.1:6837/2588582737] exists,up f0bb1a41-228d-4747-b8a8-02566db8767a 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:01:37.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:01:37.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:01:37.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:01:37.135 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:01:37.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:01:37.146 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T16:58:52.563+0000 7fb357c17640 0 load: jerasure load: lrc 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:540: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:01:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:01:38.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:01:38.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:01:39.621 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:01:39.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:01:40.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:01:40.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:01:40.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:01:40.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:01:40.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:01:40.635 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:01:40.635 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:01:40.636 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:01:40.636 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:01:40.636 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:01:40.725 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:4' 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:01:41.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:01:41.112 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:01:41.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803811 2024-09-10T17:01:41.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803811 2024-09-10T17:01:41.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811' 2024-09-10T17:01:41.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:01:41.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:01:41.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574876 2024-09-10T17:01:41.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574876 2024-09-10T17:01:41.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574876' 2024-09-10T17:01:41.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:01:41.357 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:01:41.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378644 2024-09-10T17:01:41.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378644 2024-09-10T17:01:41.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574876 2-81604378644' 2024-09-10T17:01:41.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:01:41.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:01:41.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084304 2024-09-10T17:01:41.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084304 2024-09-10T17:01:41.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574876 2-81604378644 3-120259084304' 2024-09-10T17:01:41.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:01:41.580 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:01:41.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691849 2024-09-10T17:01:41.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691849 2024-09-10T17:01:41.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803811 1-55834574876 2-81604378644 3-120259084304 4-171798691849' 2024-09-10T17:01:41.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:01:41.694 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803811 2024-09-10T17:01:41.694 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:01:41.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:01:41.695 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803811 2024-09-10T17:01:41.695 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:01:41.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803811 2024-09-10T17:01:41.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-10T17:01:41.696 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803811 2024-09-10T17:01:41.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:01:42.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-10T17:01:42.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:01:43.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:01:43.107 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:01:43.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803811 2024-09-10T17:01:43.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:01:43.533 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574876 2024-09-10T17:01:43.533 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:01:43.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:01:43.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574876 2024-09-10T17:01:43.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:01:43.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574876 2024-09-10T17:01:43.536 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574876 2024-09-10T17:01:43.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-09-10T17:01:43.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:01:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574876 -lt 55834574876 2024-09-10T17:01:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:01:43.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378644 2024-09-10T17:01:43.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:01:43.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:01:43.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378644 2024-09-10T17:01:43.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:01:43.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378644 2024-09-10T17:01:43.954 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378644 2024-09-10T17:01:43.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-10T17:01:43.954 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:01:44.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-10T17:01:44.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:01:44.361 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084304 2024-09-10T17:01:44.361 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:01:44.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:01:44.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084304 2024-09-10T17:01:44.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:01:44.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084304 2024-09-10T17:01:44.364 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084304 2024-09-10T17:01:44.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084304' 2024-09-10T17:01:44.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:01:44.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084304 -lt 120259084304 2024-09-10T17:01:44.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:01:44.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691849 2024-09-10T17:01:44.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:01:44.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:01:44.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691849 2024-09-10T17:01:44.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:01:44.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691849 2024-09-10T17:01:44.755 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691849 2024-09-10T17:01:44.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691849' 2024-09-10T17:01:44.756 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:01:45.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691849 -lt 171798691849 2024-09-10T17:01:45.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:01:45.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:01:45.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:01:45.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:01:45.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:01:45.749 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:01:45.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:01:45.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:01:45.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:01:45.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:01:45.750 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:01:46.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:01:46.126 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:01:46.126 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:01:46.126 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:01:46.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:01:46.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:01:46.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:01:46.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:01:47.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:01:47.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:01:47.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:01:47.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:01:47.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:01:47.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:01:47.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:01:47.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-10T17:01:48.047 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:01:48.047 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:41.679301+0000 0'0 48:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-10T17:01:39.074803+0000 0'0 2024-09-10T17:01:39.074803+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:53:11.805136+0000 0 0 2024-09-10T17:01:48.047 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.843184+0000 0'0 48:18 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T19:22:04.973264+0000 0 0 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.863843+0000 0'0 47:112 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:42:49.024882+0000 0 0 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.774761+0000 0'0 48:18 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:22:50.667148+0000 0 0 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:00:26.423523+0000 0'0 48:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:00:43.017298+0000 0 0 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:01:48.048 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:01:48.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:01:48.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:48.228 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-10T17:01:48.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-10T17:01:48.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-10T17:01:48.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:01:48.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T17:01:48.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: local primary=3 2024-09-10T17:01:48.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-10T17:01:48.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:01:48.613 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:01:48.613 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:01:48.613 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:01:49.025 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:01:49.025 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:01:49.025 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:01:49.025 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:01:49.025 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: local last_osd=2 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:01:49.026 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:01:49.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:01:49.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:01:49.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:01:49.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:01:49.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:01:49.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-10T17:01:50.101 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already down. 2024-09-10T17:01:50.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-10T17:01:50.881 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already out. 2024-09-10T17:01:50.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:41.679301+0000 0'0 48:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-10T17:01:39.074803+0000 0'0 2024-09-10T17:01:39.074803+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:53:11.805136+0000 0 0 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.843184+0000 0'0 48:18 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T19:22:04.973264+0000 0 0 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.863843+0000 0'0 48:114 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:42:49.024882+0000 0 0 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.774761+0000 0'0 48:18 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:22:50.667148+0000 0 0 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:00:26.423523+0000 0'0 48:74 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:00:43.017298+0000 0 0 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:01:51.254 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:01:51.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-10T17:01:51.266 INFO:tasks.workunit.client.0.smithi086.stderr:4+0 records in 2024-09-10T17:01:51.266 INFO:tasks.workunit.client.0.smithi086.stderr:4+0 records out 2024-09-10T17:01:51.266 INFO:tasks.workunit.client.0.smithi086.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8255e-05 s, 46.4 MB/s 2024-09-10T17:01:51.266 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: seq 1 300 2024-09-10T17:01:51.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:51.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:51.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:51.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:51.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:52.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:52.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:52.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:52.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:53.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:53.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:53.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:53.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:53.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:53.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:53.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:53.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:54.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:54.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:54.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:54.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:54.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:54.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:54.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:54.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:55.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:55.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:55.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:55.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:55.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:55.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:55.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:55.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:56.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:56.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:56.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:56.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:56.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:57.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:57.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:57.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:57.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:57.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:57.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:57.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:57.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:58.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:58.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:58.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:58.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:58.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:58.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:58.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:58.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:59.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:59.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:59.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:59.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:59.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:59.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-10T17:01:59.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:01:59.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:00.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:00.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:00.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:00.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:00.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:00.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:00.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:00.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:01.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:01.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:01.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:01.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:01.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:01.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:01.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:02.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:02.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:02.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:02.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:02.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:02.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:02.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:02.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:03.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:03.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:03.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:03.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:03.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:03.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:03.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:03.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:04.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:04.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:04.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:04.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:04.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:04.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:05.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:05.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:05.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:05.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:05.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:05.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:05.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:05.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:06.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:06.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:06.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:06.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:06.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:06.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:07.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:07.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:07.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:07.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:07.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:07.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:07.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:07.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:08.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:08.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:08.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:08.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:08.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:08.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:08.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:08.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:09.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:09.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:09.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:09.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:09.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:09.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:10.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:10.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:10.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:10.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:10.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:10.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:11.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:11.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:11.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:11.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:11.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:11.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:11.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:11.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:12.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:12.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:12.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:12.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:12.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:12.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:13.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:13.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:13.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:13.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:13.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:13.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:13.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:13.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:14.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:14.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:14.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:14.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:14.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:14.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:14.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:14.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:15.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:15.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:15.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:15.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:15.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:15.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:15.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:15.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:16.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:16.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:16.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:16.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:16.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:16.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:16.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:17.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:17.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:17.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:17.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:17.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:17.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:18.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:18.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:18.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:18.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:18.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:18.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:18.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:18.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:19.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:19.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:19.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:19.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:19.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:19.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:19.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:19.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:20.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:20.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:20.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:20.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:20.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:20.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:20.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:20.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:21.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:21.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:21.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:21.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:21.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:21.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:22.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:22.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:22.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:22.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:22.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:22.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:22.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:22.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:23.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:23.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:23.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:23.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:23.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:23.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:23.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:23.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:24.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:24.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:24.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:24.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:24.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:24.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:25.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:25.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:25.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:25.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:25.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:25.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:25.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:25.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:26.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:26.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:26.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:26.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:26.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:26.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:27.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:27.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:27.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:27.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:27.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:27.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:27.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:27.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:28.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:28.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:28.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:28.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:28.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:28.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:28.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:28.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:29.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:29.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:29.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:29.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:29.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:29.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:29.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:29.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:30.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:30.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:30.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:30.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:30.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:30.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:31.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:31.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:31.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:31.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:31.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:31.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:31.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:31.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:32.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:32.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:32.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:32.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:32.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:32.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:33.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:33.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:33.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:33.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:33.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:33.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:34.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:34.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:34.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:34.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:34.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:34.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:34.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:34.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:35.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:35.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:35.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:35.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:35.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:35.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:35.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:35.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:35.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:35.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:36.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:36.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:36.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:36.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:36.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:36.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:36.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:36.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:37.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:37.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:37.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:37.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:37.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:37.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:37.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:38.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:38.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:38.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:38.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:38.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:38.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:38.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:38.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:39.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:39.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:39.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:39.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:39.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:39.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:39.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:39.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:40.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:40.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:40.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:40.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:40.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:40.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:41.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:41.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:41.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:41.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:41.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:41.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:41.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:41.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:42.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:42.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:42.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:42.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:42.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:42.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:42.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:42.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:43.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:43.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:43.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:43.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:43.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:43.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:43.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:43.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:44.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:44.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:44.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:44.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:44.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:44.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:45.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:45.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:45.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:45.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:45.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:45.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:45.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:45.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:46.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:46.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:46.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:46.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:46.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:46.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:46.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:46.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:47.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:47.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:47.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:47.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:47.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:47.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:48.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:48.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:48.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:48.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:49.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:49.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:49.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:49.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:49.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:49.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:49.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:49.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:50.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:50.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:50.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:50.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:51.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:51.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:51.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:51.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:51.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:51.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:52.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:52.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:52.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:52.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:52.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:52.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:52.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:52.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:53.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:53.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:53.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:53.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:53.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:53.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:53.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:53.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:54.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:54.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:54.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:54.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:54.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:54.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:55.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:55.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:55.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:55.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:55.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:55.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:55.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:55.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:56.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:56.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:56.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:56.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:56.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:57.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:57.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:57.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:57.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:57.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:57.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:57.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:57.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:58.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:58.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:58.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:58.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:58.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:58.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:59.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:59.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:59.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:59.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:59.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:59.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-10T17:02:59.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:02:59.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:00.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:00.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:00.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:00.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:00.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:00.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:00.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:00.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:01.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:01.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:01.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:01.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:01.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:01.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:01.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:01.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:02.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:02.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:02.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:02.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:02.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:02.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:02.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:02.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:03.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:03.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:03.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:03.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:03.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:03.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:03.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:03.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:04.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:04.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:04.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:04.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:04.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:04.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:04.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:04.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:05.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:05.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:05.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:05.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:05.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:05.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:05.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:05.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:06.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:06.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:06.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:06.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:06.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:06.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:07.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:07.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:07.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:07.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:07.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:07.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:07.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:07.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:08.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:08.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:08.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:08.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:08.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:08.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:08.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:08.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:09.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:09.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:09.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:09.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:09.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:03:09.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:03:09.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-10T17:03:09.882 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:03:10.308 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:03:10.308 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:2147483647' 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:03:10.309 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:03:10.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:03:10.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:03:10.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:03:10.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:03:10.312 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:03:10.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:03:10.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:10.397 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:10.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:03:10.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:03:10.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:03:10.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj250 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj250 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2024-09-10T17:03:10.485 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-10T17:03:10.486 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:2147483647' 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:03:10.893 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:03:10.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:03:10.896 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:03:10.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:03:10.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:03:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:03:10.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-10T17:03:11.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-10T17:03:11.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:03:11.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:03:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:03:11.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:03:11.072 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:03:11.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:03:11.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:03:11.073 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:03:11.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:03:11.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:03:11.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:03:11.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:03:11.095 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:11.094+0000 7f2e387bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:11.097 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:11.096+0000 7f2e387bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:11.099 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:11.097+0000 7f2e387bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:11.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:03:11.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:03:11.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:03:11.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:03:11.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:03:11.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:11.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:03:11.456 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:03:11.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:11.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:11.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:12.679 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:12.678+0000 7f2e387bf640 -1 Falling back to public interface 2024-09-10T17:03:12.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:12.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:12.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:03:12.836 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:03:12.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:12.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:13.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:14.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:14.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:14.257 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:03:14.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:03:14.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:14.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:14.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:15.083 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:15.082+0000 7f2e387bf640 -1 osd.2 48 log_to_monitors true 2024-09-10T17:03:15.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:15.643 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:03:15.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:15.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:03:15.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:15.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:16.193 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up out weight 0 up_from 56 up_thru 19 down_at 49 last_clean_interval [19,48) [v2:127.0.0.1:6818/2900039583,v1:127.0.0.1:6819/2900039583] [v2:127.0.0.1:6820/2900039583,v1:127.0.0.1:6821/2900039583] exists,up 43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T17:03:16.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:03:16.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:03:16.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:03:16.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-10T17:03:17.551 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already in. 2024-09-10T17:03:17.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: sleep 15 2024-09-10T17:03:32.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: seq 1 240 2024-09-10T17:03:32.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:32.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:32.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:32.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:32.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:32.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:32.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:32.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:32.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:32.959 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:32.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:32.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:32.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:33.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:33.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:33.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:33.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:33.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:33.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:34.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:34.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:34.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:34.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:34.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:34.353 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:34.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:35.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:35.745 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:35.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:36.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:37.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:37.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:37.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:37.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:37.159 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:37.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:37.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:38.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:38.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:38.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:38.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:38.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:38.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:38.546 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:38.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:39.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:39.548 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:39.548 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:39.548 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:39.549 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:39.549 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:39.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:39.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:39.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:39.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:39.939 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:39.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:39.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:40.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:41.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:41.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-10T17:03:41.316 INFO:tasks.workunit.client.0.smithi086.stdout:active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-10T17:03:41.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: sleep 1 2024-09-10T17:03:42.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-10T17:03:42.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2024-09-10T17:03:42.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:03:42.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:03:42.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:03:42.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:03:42.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-10T17:03:42.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-10T17:03:42.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-10T17:03:42.704 INFO:tasks.workunit.client.0.smithi086.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-10T17:03:42.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-10T17:03:42.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: break 2024-09-10T17:03:42.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-10T17:03:43.066 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-10T17:03:40.535319+0000 54'301 62:903 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-10T17:01:39.074803+0000 0'0 2024-09-10T17:01:39.074803+0000 0 0 no scrub is scheduled 0 0 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:03:18.521080+0000 0'0 62:61 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T19:59:53.456515+0000 0 0 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.863843+0000 0'0 62:141 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:42:49.024882+0000 0 0 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.774761+0000 0'0 62:44 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:22:50.667148+0000 0 0 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:03:18.520871+0000 0'0 62:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:41:50.675657+0000 0 0 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:03:43.067 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:03:43.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:583: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:03:43.077 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:03:43.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:03:43.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:03:43.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:03:43.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:03:43.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:03:43.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: sleep 5 2024-09-10T17:03:48.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:586: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-10T17:03:44.511684+0000 54'301 64:915 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-10T17:01:39.074803+0000 0'0 2024-09-10T17:01:39.074803+0000 0 0 no scrub is scheduled 0 0 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-10T17:03:44.402808+0000 0'0 64:70 [4,1] 4 [4,1] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T18:08:24.358334+0000 0 0 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.863843+0000 0'0 64:145 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:42:49.024882+0000 0 0 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:01:39.774761+0000 0'0 64:49 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:22:50.667148+0000 0 0 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-10T17:03:44.402813+0000 0'0 64:123 [1,0] 1 [1,0] 1 0'0 2024-09-10T16:58:42.880248+0000 0'0 2024-09-10T16:58:42.880248+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:36:25.717496+0000 0 0 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:03:48.754 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:03:48.755 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:03:48.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:03:48.918 INFO:tasks.workunit.client.0.smithi086.stdout:{ 2024-09-10T17:03:48.918 INFO:tasks.workunit.client.0.smithi086.stdout: "num_missing": 1, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "num_unfound": 1, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": [ 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "oid": { 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "oid": "obj250", 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "key": "", 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "snapid": -2, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "hash": 2249616407, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "max": 0, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "pool": 2, 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: "namespace": "" 2024-09-10T17:03:48.919 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "need": "54'251", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "have": "0'0", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "flags": "none", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "locations": [ 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: ] 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "NotRecovering", 2024-09-10T17:03:48.920 INFO:tasks.workunit.client.0.smithi086.stdout: "available_might_have_unfound": true, 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: "might_have_unfound": [ 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "2(4)", 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "osd is down" 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout: "more": false 2024-09-10T17:03:48.921 INFO:tasks.workunit.client.0.smithi086.stdout:} 2024-09-10T17:03:48.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:588: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout:{ 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: "snap_trimq": "[]", 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: "snap_trimq_len": 0, 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch": 64, 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.051 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_recovery_backfill": [ 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:03:49.052 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "info": { 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s0", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "0", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "54'301", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "54'250", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "54'200", 2024-09-10T17:03:49.053 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 301, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 46, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 46, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 63, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 46, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:03:49.054 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 63, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 63, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 46, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "54'301", 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 916, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 64, 2024-09-10T17:03:49.055 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:03:44.511684+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:03:44.511684+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:03:44.511684+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:03:44.511684+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:01:48.224973+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:03:44.487138+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:03:44.487138+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:03:44.511684+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:03:44.125460+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:03:44.125262+0000", 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 63, 2024-09-10T17:03:49.056 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "54'200", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "54'200", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 46, 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:03:49.057 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 200, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:03:49.058 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 1232896, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 301, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 1505, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 1, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 1, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 303, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 1, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 301, 2024-09-10T17:03:49.059 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 301, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 1204, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:03:49.060 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:03:49.061 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.062 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:03:49.063 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 300 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),4(2)", 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 1 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:03:49.064 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "peer_info": [ 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "0(3)", 2024-09-10T17:03:49.065 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s3", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "3", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "54'301", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "54'301", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "54'200", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 301, 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 46, 2024-09-10T17:03:49.066 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 46, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 63, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 46, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 63, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 63, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 46, 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.067 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "54'301", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 906, 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 63, 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "activating+undersized+degraded+remapped", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:03:18.906051+0000", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:03:16.053816+0000", 2024-09-10T17:03:49.068 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:03:15.634971+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:01:48.224973+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:03:18.746968+0000", 2024-09-10T17:03:49.069 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:03:18.746506+0000", 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 63, 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "54'200", 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "54'200", 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 46, 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.070 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.071 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 200, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:03:49.072 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:03:49.073 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 1232896, 2024-09-10T17:03:49.074 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 301, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 1505, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 0, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 301, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:03:49.075 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 301, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 301, 2024-09-10T17:03:49.076 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 1204, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:03:49.077 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:03:49.078 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:03:49.079 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:03:49.080 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.081 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.082 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:03:49.083 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 301 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.084 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:03:49.085 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:03:49.086 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "1(1)", 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s1", 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "1", 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "54'301", 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "54'301", 2024-09-10T17:03:49.087 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "54'200", 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 301, 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 46, 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 46, 2024-09-10T17:03:49.088 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 63, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 46, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 63, 2024-09-10T17:03:49.089 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 63, 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 46, 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.090 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "54'301", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 349, 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 62, 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "activating+undersized+degraded+remapped", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:03:18.906051+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:03:16.053816+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:03:15.634971+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:01:48.224973+0000", 2024-09-10T17:03:49.091 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:03:18.746968+0000", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:03:18.746506+0000", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 63, 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "54'200", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "54'200", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 46, 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.092 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 200, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:03:49.093 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 1232896, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 301, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 1505, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:03:49.094 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 1, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 301, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 301, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 301, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 1204, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:03:49.095 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:03:49.096 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.097 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.098 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 301 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.099 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.100 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "4(2)", 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s2", 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "2", 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "54'301", 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "54'301", 2024-09-10T17:03:49.101 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "54'200", 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 301, 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 46, 2024-09-10T17:03:49.102 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 46, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 63, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 46, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 63, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 63, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 46, 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.103 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "54'301", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 349, 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 62, 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "activating+undersized+degraded+remapped", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:03:18.906051+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:03:16.053816+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:03:15.634971+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:01:48.224973+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.104 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:01:50.487637+0000", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:03:19.157572+0000", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:03:18.746968+0000", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:03:18.746506+0000", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 63, 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "54'200", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "54'200", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 46, 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 47, 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:03:49.105 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:01:39.074803+0000", 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 200, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:03:49.106 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 1232896, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 301, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 1505, 2024-09-10T17:03:49.107 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 301, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 301, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 301, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 1204, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:03:49.108 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:03:49.109 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.110 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: 2147483647 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.111 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 301 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.112 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 64, 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.113 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_state": [ 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "name": "Started/Primary/Active", 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "enter_time": "2024-09-10T17:03:44.125294+0000", 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "might_have_unfound": [ 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "0(3)", 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "1(1)", 2024-09-10T17:03:49.114 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "2(4)", 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "osd is down" 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "4(2)", 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_progress": { 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "backfill_targets": [], 2024-09-10T17:03:49.115 INFO:tasks.workunit.client.0.smithi086.stdout: "waiting_on_backfill": [], 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill_started": "MIN", 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "backfill_info": { 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "begin": "MIN", 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "end": "MIN", 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": [] 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "peer_backfill_info": [], 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "backfills_in_flight": [], 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "recovering": [], 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "pg_backend": { 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_ops": [], 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: "read_ops": [] 2024-09-10T17:03:49.116 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "name": "Started", 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "enter_time": "2024-09-10T17:03:43.455840+0000" 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "scrubber": { 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "active": false, 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "must_scrub": false, 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "must_deep_scrub": false, 2024-09-10T17:03:49.117 INFO:tasks.workunit.client.0.smithi086.stdout: "must_repair": false, 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout: "need_auto": false, 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_reg_stamp": "2024-09-12T03:32:52.822318+0000", 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout: "schedule": "not registered for scrubbing" 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout: "agent_state": {} 2024-09-10T17:03:49.118 INFO:tasks.workunit.client.0.smithi086.stdout:} 2024-09-10T17:03:49.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:03:49.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: grep -q obj250 2024-09-10T17:03:49.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:03:49.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-10T17:03:49.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: check=true 2024-09-10T17:03:49.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: test true == true 2024-09-10T17:03:49.295 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:03:49.295 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-10T17:03:49.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-10T17:03:49.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: check='osd is down' 2024-09-10T17:03:49.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-10T17:03:49.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:03:49.412 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: check='2(4)' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:03:49.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:03:49.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:03:49.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:03:49.533 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:03:49.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:03:49.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:03:49.534 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:03:49.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:03:49.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:03:49.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:03:49.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:03:49.556 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:49.555+0000 7f4207211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:49.558 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:49.557+0000 7f4207211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:49.560 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:49.558+0000 7f4207211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:03:49.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:03:49.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:03:49.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:03:49.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:03:49.944 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:03:49.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:03:49.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:49.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:03:49.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:49.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:50.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:51.154 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:51.152+0000 7f4207211640 -1 Falling back to public interface 2024-09-10T17:03:51.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:51.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:51.391 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:03:51.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:03:51.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:51.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:51.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:52.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:52.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:52.818 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:03:52.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:03:52.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:52.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:53.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:03:53.329 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:03:53.327+0000 7f4207211640 -1 osd.2 62 log_to_monitors true 2024-09-10T17:03:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:03:54.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:03:54.207 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:03:54.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:03:54.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:03:54.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:03:54.738 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 66 up_thru 19 down_at 63 last_clean_interval [56,62) [v2:127.0.0.1:6818/1959307089,v1:127.0.0.1:6819/1959307089] [v2:127.0.0.1:6820/1959307089,v1:127.0.0.1:6821/1959307089] exists,up 43a9a5a1-4741-41af-8338-5d68efcbb7df 2024-09-10T17:03:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:03:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:03:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:03:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-10T17:03:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:605: TEST_ec_backfill_unfound: test 124 = 124 2024-09-10T17:03:59.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:607: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-10T17:03:59.994 INFO:tasks.workunit.client.0.smithi086.stderr:pg has no unfound objects 2024-09-10T17:04:00.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_backfill_unfound: wait_for_clean 2024-09-10T17:04:00.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:04:00.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:04:00.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:04:00.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:04:00.005 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:04:00.005 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:04:00.005 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:04:00.005 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:04:00.005 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:04:00.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:04:00.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:04:00.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:04:00.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:04:00.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:04:00.093 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:04:00.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:04:00.480 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:04:00.480 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:04:00.480 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:04:00.481 INFO:tasks.workunit.client.0.smithi086.stderr:4' 2024-09-10T17:04:00.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:04:00.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:04:00.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:04:00.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803841 2024-09-10T17:04:00.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803841 2024-09-10T17:04:00.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841' 2024-09-10T17:04:00.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:04:00.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:04:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574905 2024-09-10T17:04:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574905 2024-09-10T17:04:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574905' 2024-09-10T17:04:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:04:00.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:04:00.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841540 2024-09-10T17:04:00.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841540 2024-09-10T17:04:00.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-283467841540' 2024-09-10T17:04:00.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:04:00.930 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:04:01.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084334 2024-09-10T17:04:01.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084334 2024-09-10T17:04:01.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-283467841540 3-120259084334' 2024-09-10T17:04:01.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:04:01.055 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:04:01.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691878 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691878 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-283467841540 3-120259084334 4-171798691878' 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803841 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:04:01.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:04:01.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803841 2024-09-10T17:04:01.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:04:01.171 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803841 2024-09-10T17:04:01.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803841 2024-09-10T17:04:01.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-10T17:04:01.171 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:04:01.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-10T17:04:01.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:04:01.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574905 2024-09-10T17:04:01.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:04:01.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:04:01.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574905 2024-09-10T17:04:01.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:04:01.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574905 2024-09-10T17:04:01.561 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574905 2024-09-10T17:04:01.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574905' 2024-09-10T17:04:01.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:04:02.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574903 -lt 55834574905 2024-09-10T17:04:02.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:04:03.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:04:03.039 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:04:03.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574906 -lt 55834574905 2024-09-10T17:04:03.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:04:03.637 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-283467841540 2024-09-10T17:04:03.637 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:04:03.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:04:03.638 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-283467841540 2024-09-10T17:04:03.638 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:04:03.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841540 2024-09-10T17:04:03.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 283467841540' 2024-09-10T17:04:03.640 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 283467841540 2024-09-10T17:04:03.640 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:04:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841540 -lt 283467841540 2024-09-10T17:04:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:04:04.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084334 2024-09-10T17:04:04.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:04:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:04:04.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084334 2024-09-10T17:04:04.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:04:04.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084334 2024-09-10T17:04:04.349 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084334 2024-09-10T17:04:04.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084334' 2024-09-10T17:04:04.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:04:04.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084334 -lt 120259084334 2024-09-10T17:04:04.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:04:04.903 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691878 2024-09-10T17:04:04.903 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:04:04.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:04:04.904 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691878 2024-09-10T17:04:04.904 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:04:04.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691878 2024-09-10T17:04:04.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691878' 2024-09-10T17:04:04.905 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691878 2024-09-10T17:04:04.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:04:05.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691879 -lt 171798691878 2024-09-10T17:04:05.284 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:04:05.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:04:05.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:04:05.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:04:05.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:04:05.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:04:05.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:04:05.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:04:05.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:04:05.809 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:04:05.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:04:06.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:04:06.215 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:04:06.215 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:04:06.215 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:04:06.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:04:06.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T17:04:06.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:04:06.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T17:04:06.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:04:06.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:04:06.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:04:06.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:04:07.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:04:07.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:04:07.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:04:07.206 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:04:07.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:04:07.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:04:07.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:04:07.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: seq 1 300 2024-09-10T17:04:07.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-10T17:04:07.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-10T17:04:07.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-10T17:04:07.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-10T17:04:07.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-10T17:04:07.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-10T17:04:07.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-10T17:04:07.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-10T17:04:07.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:07.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:07.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-10T17:04:07.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-10T17:04:08.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-10T17:04:08.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-10T17:04:08.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-10T17:04:08.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-10T17:04:08.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-10T17:04:08.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-10T17:04:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-10T17:04:08.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-10T17:04:08.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-10T17:04:08.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-10T17:04:08.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-10T17:04:08.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-10T17:04:08.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-10T17:04:08.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-10T17:04:08.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-10T17:04:08.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-10T17:04:08.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-10T17:04:08.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-10T17:04:08.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-10T17:04:08.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-10T17:04:08.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-10T17:04:08.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-10T17:04:08.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-10T17:04:08.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-10T17:04:08.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-10T17:04:08.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-10T17:04:08.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-10T17:04:08.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-10T17:04:08.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-10T17:04:08.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:08.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:08.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-10T17:04:08.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-10T17:04:09.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-10T17:04:09.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-10T17:04:09.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-10T17:04:09.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-10T17:04:09.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-10T17:04:09.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-10T17:04:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-10T17:04:09.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-10T17:04:09.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-10T17:04:09.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-10T17:04:09.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-10T17:04:09.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-10T17:04:09.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-10T17:04:09.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-10T17:04:09.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-10T17:04:09.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-10T17:04:09.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-10T17:04:09.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-10T17:04:09.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-10T17:04:09.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-10T17:04:09.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-10T17:04:09.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-10T17:04:09.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-10T17:04:09.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-10T17:04:09.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-10T17:04:09.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-10T17:04:09.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-10T17:04:09.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-10T17:04:09.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-10T17:04:09.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-10T17:04:09.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-10T17:04:09.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:09.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:09.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-10T17:04:09.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-10T17:04:10.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-10T17:04:10.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-10T17:04:10.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-10T17:04:10.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-10T17:04:10.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-10T17:04:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-10T17:04:10.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-10T17:04:10.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-10T17:04:10.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-10T17:04:10.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-10T17:04:10.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-10T17:04:10.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-10T17:04:10.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-10T17:04:10.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-10T17:04:10.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-10T17:04:10.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-10T17:04:10.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-10T17:04:10.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-10T17:04:10.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-10T17:04:10.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-10T17:04:10.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-10T17:04:10.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-10T17:04:10.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-10T17:04:10.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-10T17:04:10.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-10T17:04:10.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-10T17:04:10.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-10T17:04:10.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-10T17:04:10.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:10.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:10.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-10T17:04:10.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-10T17:04:11.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-10T17:04:11.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-10T17:04:11.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-10T17:04:11.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-10T17:04:11.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-10T17:04:11.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-10T17:04:11.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-10T17:04:11.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-10T17:04:11.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-10T17:04:11.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-10T17:04:11.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-10T17:04:11.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-10T17:04:11.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-10T17:04:11.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-10T17:04:11.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-10T17:04:11.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-10T17:04:11.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-10T17:04:11.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-10T17:04:11.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-10T17:04:11.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-10T17:04:11.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-10T17:04:11.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-10T17:04:11.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-10T17:04:11.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-10T17:04:11.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-10T17:04:11.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-10T17:04:11.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-10T17:04:11.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-10T17:04:11.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-10T17:04:11.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-10T17:04:11.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-10T17:04:11.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:11.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:11.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-10T17:04:11.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-10T17:04:12.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-10T17:04:12.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-10T17:04:12.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-10T17:04:12.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-10T17:04:12.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-10T17:04:12.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-10T17:04:12.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-10T17:04:12.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-10T17:04:12.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-10T17:04:12.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-10T17:04:12.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-10T17:04:12.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-10T17:04:12.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-10T17:04:12.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-10T17:04:12.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-10T17:04:12.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-10T17:04:12.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-10T17:04:12.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-10T17:04:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-10T17:04:12.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-10T17:04:12.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-10T17:04:12.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-10T17:04:12.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-10T17:04:12.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-10T17:04:12.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-10T17:04:12.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-10T17:04:12.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-10T17:04:12.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-10T17:04:12.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-10T17:04:12.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:12.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:12.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-10T17:04:12.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-10T17:04:13.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-10T17:04:13.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-10T17:04:13.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-10T17:04:13.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-10T17:04:13.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-10T17:04:13.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-10T17:04:13.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-10T17:04:13.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-10T17:04:13.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-10T17:04:13.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-10T17:04:13.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-10T17:04:13.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-10T17:04:13.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-10T17:04:13.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-10T17:04:13.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-10T17:04:13.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-10T17:04:13.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-10T17:04:13.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-10T17:04:13.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-10T17:04:13.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-10T17:04:13.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-10T17:04:13.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-10T17:04:13.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-10T17:04:13.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-10T17:04:13.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-10T17:04:13.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-10T17:04:13.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-10T17:04:13.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-10T17:04:13.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-10T17:04:13.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-10T17:04:13.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-10T17:04:13.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:13.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:13.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-10T17:04:13.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-10T17:04:14.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-10T17:04:14.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-10T17:04:14.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-10T17:04:14.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-10T17:04:14.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-10T17:04:14.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-10T17:04:14.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-10T17:04:14.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-10T17:04:14.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-10T17:04:14.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-10T17:04:14.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-10T17:04:14.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-10T17:04:14.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-10T17:04:14.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-10T17:04:14.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-10T17:04:14.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-10T17:04:14.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-10T17:04:14.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-10T17:04:14.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-10T17:04:14.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-10T17:04:14.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-10T17:04:14.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-10T17:04:14.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-10T17:04:14.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-10T17:04:14.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-10T17:04:14.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-10T17:04:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-10T17:04:14.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-10T17:04:14.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:14.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:14.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-10T17:04:14.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-10T17:04:15.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-10T17:04:15.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-10T17:04:15.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-10T17:04:15.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-10T17:04:15.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-10T17:04:15.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-10T17:04:15.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-10T17:04:15.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-10T17:04:15.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-10T17:04:15.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-10T17:04:15.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-10T17:04:15.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-10T17:04:15.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-10T17:04:15.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-10T17:04:15.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-10T17:04:15.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-10T17:04:15.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-10T17:04:15.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-10T17:04:15.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-10T17:04:15.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-10T17:04:15.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-10T17:04:15.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-10T17:04:15.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-10T17:04:15.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-10T17:04:15.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-10T17:04:15.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-10T17:04:15.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-10T17:04:15.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.870 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-10T17:04:15.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-10T17:04:15.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-10T17:04:15.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-10T17:04:15.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-10T17:04:15.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-10T17:04:15.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:15.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:15.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-10T17:04:15.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-10T17:04:16.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-10T17:04:16.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-10T17:04:16.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-10T17:04:16.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-10T17:04:16.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-10T17:04:16.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-10T17:04:16.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-10T17:04:16.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-10T17:04:16.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-10T17:04:16.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-10T17:04:16.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-10T17:04:16.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-10T17:04:16.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-10T17:04:16.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-10T17:04:16.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-10T17:04:16.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-10T17:04:16.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-10T17:04:16.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-10T17:04:16.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-10T17:04:16.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-10T17:04:16.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-10T17:04:16.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-10T17:04:16.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-10T17:04:16.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-10T17:04:16.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-10T17:04:16.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-10T17:04:16.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-10T17:04:16.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-10T17:04:16.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-10T17:04:16.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:16.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:16.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-10T17:04:16.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-10T17:04:17.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-10T17:04:17.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-10T17:04:17.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-10T17:04:17.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-10T17:04:17.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-10T17:04:17.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-10T17:04:17.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-10T17:04:17.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-10T17:04:17.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-10T17:04:17.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-10T17:04:17.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-10T17:04:17.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-10T17:04:17.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-10T17:04:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:04:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-10T17:04:17.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-10T17:04:17.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:04:17.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:04:17.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:04:17.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:04:17.972 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:04:17.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:04:18.759 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:04:18.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:04:18.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:04:18.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:04:18.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:04:18.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:04:18.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:04:18.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:04:18.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:04:18.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:04:18.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:04:18.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:04:18.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:04:18.884 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:04:18.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:04:18.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:04:18.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:04:18.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:04:18.944 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:04:18.944 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:18.944 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:18.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:04:18.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:04:18.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:04:18.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:04:18.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:04:18.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:04:18.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:04:18.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:04:18.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:04:18.948 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:04:18.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:04:18.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:04:18.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:04:18.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:04:18.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:18.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:18.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:04:18.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:04:18.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:04:18.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:04:18.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:04:18.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:18.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:18.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:04:18.955 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:04:18.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:04:18.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:04:18.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:04:18.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:04:18.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:04:19.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:04:19.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:04:19.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:04:19.419 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:04:19.420 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:19.420 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:19.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:19.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:04:19.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:04:19.612 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:04:19.613 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:04:19.613 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:04:19.613 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:19.614 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:19.614 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:04:19.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:04:19.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:04:19.698 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:19.699 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:19.699 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:04:19.699 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:04:19.699 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:04:19.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:04:19.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:04:20.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:04:20.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:20.626 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:04:20.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:04:20.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:04:20.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:04:20.943 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:04:20.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:04:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:04:22.029 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:04:22.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:04:22.038 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:04:19.453+0000 7f4cdbf22c80 0 load: jerasure load: lrc 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:379: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-10T17:04:22.039 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:04:22.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:04:22.045 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:04:22.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=99427ef0-4650-4c01-ae2a-982bfd96a12b 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 99427ef0-4650-4c01-ae2a-982bfd96a12b' 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 99427ef0-4650-4c01-ae2a-982bfd96a12b 2024-09-10T17:04:22.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:04:22.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWfOBmLqWcAxAAg0UVkpBJlJk6JXFBmCQGiA== 2024-09-10T17:04:22.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWfOBmLqWcAxAAg0UVkpBJlJk6JXFBmCQGiA=="}' 2024-09-10T17:04:22.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 99427ef0-4650-4c01-ae2a-982bfd96a12b -i td/test-erasure-eio/0/new.json 2024-09-10T17:04:22.320 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:04:22.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:04:22.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWfOBmLqWcAxAAg0UVkpBJlJk6JXFBmCQGiA== --osd-uuid 99427ef0-4650-4c01-ae2a-982bfd96a12b 2024-09-10T17:04:22.359 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:22.358+0000 7f315140e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:22.380 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:22.379+0000 7f315140e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:22.381 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:22.380+0000 7f315140e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:22.382 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:22.380+0000 7f315140e640 -1 bdev(0x5640bc7ec800 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:04:22.382 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:22.381+0000 7f315140e640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:04:28.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:04:28.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:04:28.793 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:04:28.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:04:28.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:04:29.329 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:04:29.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:04:29.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:04:29.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:04:29.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:04:29.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:04:29.356 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:29.355+0000 7f1c5200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:29.376 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:29.375+0000 7f1c5200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:29.377 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:29.376+0000 7f1c5200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:29.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:04:29.782 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:04:29.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:29.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:30.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:30.969 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:30.967+0000 7f1c5200f640 -1 Falling back to public interface 2024-09-10T17:04:31.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:31.210 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:04:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:04:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:31.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:31.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:32.528 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:32.527+0000 7f1c5200f640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:04:32.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:32.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:32.643 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:04:32.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:04:32.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:32.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:33.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:34.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:34.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:34.115 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:04:34.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:04:34.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:34.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:34.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:35.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:35.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:35.612 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:04:35.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:04:35.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:35.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:36.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:37.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:37.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:37.036 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:04:37.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:04:37.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:37.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:37.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:38.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:38.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:38.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:04:38.449 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:04:38.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:38.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:38.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:39.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:39.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:39.846 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:04:39.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:04:39.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:39.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:40.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:41.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:41.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:41.240 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:04:41.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:04:41.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:41.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:41.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:42.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:42.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:42.650 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:04:42.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:04:42.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:42.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:43.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:44.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:44.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:44.063 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:04:44.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:04:44.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:44.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:44.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:45.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:45.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:45.821 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:04:45.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:04:45.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:45.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:46.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:47.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:47.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:47.353 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:04:47.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:04:47.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:47.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:48.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:49.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:49.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:49.376 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:04:49.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:04:49.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:49.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:49.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:50.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:50.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:50.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:04:50.912 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:04:50.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:50.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:51.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:52.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:52.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:52.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:04:52.874 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:04:52.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:52.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:53.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:54.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:54.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:54.670 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:04:54.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:04:54.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:54.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:55.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:04:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:04:56.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:04:56.182 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:04:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:04:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:04:56.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:04:56.608 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/777630752,v1:127.0.0.1:6803/777630752] [v2:127.0.0.1:6804/777630752,v1:127.0.0.1:6805/777630752] exists,up 99427ef0-4650-4c01-ae2a-982bfd96a12b 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:04:56.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:04:56.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:04:56.616 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:04:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:04:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36623254-1536-4cee-8c72-6a3c0ed27ece 2024-09-10T17:04:56.619 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 36623254-1536-4cee-8c72-6a3c0ed27ece 2024-09-10T17:04:56.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 36623254-1536-4cee-8c72-6a3c0ed27ece' 2024-09-10T17:04:56.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:04:56.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4fOBmP6F7JRAAbdx1QA4z2OVDq2btJVOldQ== 2024-09-10T17:04:56.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4fOBmP6F7JRAAbdx1QA4z2OVDq2btJVOldQ=="}' 2024-09-10T17:04:56.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36623254-1536-4cee-8c72-6a3c0ed27ece -i td/test-erasure-eio/1/new.json 2024-09-10T17:04:57.174 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:04:57.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:04:57.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4fOBmP6F7JRAAbdx1QA4z2OVDq2btJVOldQ== --osd-uuid 36623254-1536-4cee-8c72-6a3c0ed27ece 2024-09-10T17:04:57.215 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:57.213+0000 7fc7ebe20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:57.216 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:57.215+0000 7fc7ebe20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:57.217 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:57.216+0000 7fc7ebe20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:04:57.218 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:57.217+0000 7fc7ebe20640 -1 bdev(0x56497c683c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:04:57.218 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:04:57.217+0000 7fc7ebe20640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:05:03.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:05:03.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:05:03.543 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:05:03.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:05:03.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:05:04.074 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:05:04.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:05:04.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:05:04.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:05:04.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:05:04.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:05:04.101 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:04.100+0000 7ff6fba62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:04.118 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:04.117+0000 7ff6fba62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:04.119 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:04.118+0000 7ff6fba62640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:04.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:05:04.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:05:04.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:05:04.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:05:04.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:05:04.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:04.533 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:05:04.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:05:04.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:04.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:04.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:05.472 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:05.470+0000 7ff6fba62640 -1 Falling back to public interface 2024-09-10T17:05:05.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:05.985 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:05:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:05:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:06.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:07.147 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:07.146+0000 7ff6fba62640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:05:07.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:07.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:07.382 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:05:07.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:05:07.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:07.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:07.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:08.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:08.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:08.851 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:05:08.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:05:08.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:08.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:09.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:10.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:10.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:10.364 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:05:10.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:05:10.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:10.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:10.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:11.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:11.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:11.762 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:05:11.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:05:11.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:11.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:12.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:13.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:13.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:13.284 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:05:13.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:05:13.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:13.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:13.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:14.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:14.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:14.692 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:05:14.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:05:14.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:14.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:15.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:16.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:16.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:16.188 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:05:16.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:05:16.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:16.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:16.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:17.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:17.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:17.586 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:05:17.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:05:17.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:17.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:18.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:19.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:19.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:19.108 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:05:19.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:05:19.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:19.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:19.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:20.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:20.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:20.755 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:05:20.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:05:20.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:20.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:21.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:22.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:22.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:22.952 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:05:22.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:05:22.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:22.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:23.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:24.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:24.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:24.434 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:05:24.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:05:24.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:24.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:24.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:25.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:25.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:25.888 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:05:25.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:05:25.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:25.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:26.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:27.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:27.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:27.791 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:05:27.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:05:27.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:27.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:28.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:29.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:29.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:29.611 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:05:29.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:05:29.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:29.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:05:30.065 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2772248730,v1:127.0.0.1:6811/2772248730] [v2:127.0.0.1:6812/2772248730,v1:127.0.0.1:6813/2772248730] exists,up 36623254-1536-4cee-8c72-6a3c0ed27ece 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:05:30.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:05:30.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:05:30.071 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:05:30.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43389017-2df0-40d2-afe7-fc7d144dd9d6 2024-09-10T17:05:30.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 43389017-2df0-40d2-afe7-fc7d144dd9d6' 2024-09-10T17:05:30.074 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 43389017-2df0-40d2-afe7-fc7d144dd9d6 2024-09-10T17:05:30.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:05:30.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBafOBmZ1UwBRAAQ5VHLvBi67BLAmSA8mJYUg== 2024-09-10T17:05:30.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBafOBmZ1UwBRAAQ5VHLvBi67BLAmSA8mJYUg=="}' 2024-09-10T17:05:30.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43389017-2df0-40d2-afe7-fc7d144dd9d6 -i td/test-erasure-eio/2/new.json 2024-09-10T17:05:30.756 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:05:30.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:05:30.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBafOBmZ1UwBRAAQ5VHLvBi67BLAmSA8mJYUg== --osd-uuid 43389017-2df0-40d2-afe7-fc7d144dd9d6 2024-09-10T17:05:30.797 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:30.795+0000 7fc83ae0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:30.900 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:30.898+0000 7fc83ae0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:30.901 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:30.899+0000 7fc83ae0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:30.901 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:30.900+0000 7fc83ae0e640 -1 bdev(0x55e6bea73c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:05:30.901 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:30.900+0000 7fc83ae0e640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:05:36.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:05:36.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:05:36.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:05:36.653 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:05:36.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:05:37.187 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:05:37.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:05:37.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:05:37.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:05:37.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:05:37.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:05:37.214 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:37.213+0000 7f88fc60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:37.237 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:37.235+0000 7f88fc60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:37.237 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:37.236+0000 7f88fc60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:37.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:05:37.635 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:05:37.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:37.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:38.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:38.581 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:38.580+0000 7f88fc60b640 -1 Falling back to public interface 2024-09-10T17:05:39.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:39.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:39.086 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:05:39.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:05:39.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:39.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:39.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:40.166 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:05:40.163+0000 7f88fc60b640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:05:40.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:40.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:40.472 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:05:40.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:05:40.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:40.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:41.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:42.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:42.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:42.085 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:05:42.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:05:42.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:42.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:42.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:43.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:43.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:43.482 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:05:43.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:05:43.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:43.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:43.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:44.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:44.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:44.981 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:05:44.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:05:44.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:44.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:45.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:46.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:46.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:46.452 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:05:46.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:05:46.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:46.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:46.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:47.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:47.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:47.866 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:05:47.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:05:47.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:47.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:48.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:49.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:49.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:49.359 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:05:49.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:05:49.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:49.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:49.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:50.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:50.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:50.756 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:05:50.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:05:50.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:50.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:51.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:52.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:52.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:52.174 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:05:52.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:05:52.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:52.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:52.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:53.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:53.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:53.670 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:05:53.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:05:53.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:53.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:54.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:55.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:55.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:55.365 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:05:55.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:05:55.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:55.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:56.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:57.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:57.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:57.326 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:05:57.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:05:57.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:57.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:58.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:05:59.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:05:59.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:05:59.242 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:05:59.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:05:59.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:05:59.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:05:59.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:00.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:00.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:00.808 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:06:00.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:06:00.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:00.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:06:01.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:02.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:02.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:02.269 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:06:02.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:06:02.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:02.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:06:02.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:03.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:03.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:03.934 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:06:03.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:06:03.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:03.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:06:04.340 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3344494836,v1:127.0.0.1:6819/3344494836] [v2:127.0.0.1:6820/3344494836,v1:127.0.0.1:6821/3344494836] exists,up 43389017-2df0-40d2-afe7-fc7d144dd9d6 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:06:04.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:06:04.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:06:04.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2ea8baa6-9832-4e43-bd27-5e579581e7e9 2024-09-10T17:06:04.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2ea8baa6-9832-4e43-bd27-5e579581e7e9' 2024-09-10T17:06:04.349 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 2ea8baa6-9832-4e43-bd27-5e579581e7e9 2024-09-10T17:06:04.349 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:06:04.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8fOBmVKyTFRAA8T8bnlyxCpKLHPArRuq2Ng== 2024-09-10T17:06:04.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8fOBmVKyTFRAA8T8bnlyxCpKLHPArRuq2Ng=="}' 2024-09-10T17:06:04.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2ea8baa6-9832-4e43-bd27-5e579581e7e9 -i td/test-erasure-eio/3/new.json 2024-09-10T17:06:05.187 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:06:05.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:06:05.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8fOBmVKyTFRAA8T8bnlyxCpKLHPArRuq2Ng== --osd-uuid 2ea8baa6-9832-4e43-bd27-5e579581e7e9 2024-09-10T17:06:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:05.227+0000 7f37d85c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:05.230 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:05.229+0000 7f37d85c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:05.232 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:05.230+0000 7f37d85c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:05.232 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:05.230+0000 7f37d85c5640 -1 bdev(0x55f200e9fc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:06:05.232 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:05.230+0000 7f37d85c5640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:06:11.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:06:11.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:06:11.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:06:11.124 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:06:11.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:06:11.655 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:06:11.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:06:11.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:06:11.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:06:11.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:06:11.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:06:11.684 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:11.682+0000 7fe58dc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:11.711 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:11.709+0000 7fe58dc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:11.712 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:11.710+0000 7fe58dc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:12.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:06:12.117 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:06:12.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:12.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:12.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:12.795 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:12.794+0000 7fe58dc0f640 -1 Falling back to public interface 2024-09-10T17:06:13.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:13.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:13.501 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:06:13.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:06:13.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:13.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:13.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:14.314 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:14.312+0000 7fe58dc0f640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:06:14.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:14.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:14.899 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:06:14.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:06:14.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:15.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:16.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:16.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:16.645 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:06:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:06:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:17.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:18.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:18.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:18.332 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:06:18.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:06:18.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:18.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:18.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:19.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:19.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:19.769 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:06:19.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:06:19.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:19.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:20.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:21.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:21.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:21.277 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:06:21.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:06:21.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:21.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:21.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:22.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:22.672 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:06:22.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:22.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:06:22.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:22.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:23.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:24.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:24.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:24.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:06:24.232 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:06:24.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:24.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:24.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:25.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:25.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:25.660 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:06:25.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:06:25.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:25.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:26.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:27.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:27.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:27.160 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:06:27.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:06:27.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:27.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:27.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:28.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:28.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:28.932 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:06:28.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:06:28.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:28.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:30.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:31.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:31.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:31.093 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:06:31.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:06:31.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:31.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:31.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:32.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:32.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:32.799 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:06:32.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:06:32.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:32.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:33.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:34.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:34.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:34.382 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:06:34.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:06:34.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:34.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:35.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:36.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:36.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:36.271 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:06:36.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:06:36.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:36.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:37.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:38.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:38.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:38.056 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:06:38.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:06:38.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:38.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:06:38.517 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2308895548,v1:127.0.0.1:6827/2308895548] [v2:127.0.0.1:6828/2308895548,v1:127.0.0.1:6829/2308895548] exists,up 2ea8baa6-9832-4e43-bd27-5e579581e7e9 2024-09-10T17:06:38.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:06:38.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:06:38.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:06:38.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:06:38.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:06:38.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:06:38.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:06:38.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:06:38.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:06:38.527 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:06:38.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:06:38.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:06:38.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:06:38.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:06:38.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:06:38.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:06:38.530 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 0ace607f-4a34-4b6c-9963-39037a8d7523 2024-09-10T17:06:38.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0ace607f-4a34-4b6c-9963-39037a8d7523 2024-09-10T17:06:38.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 0ace607f-4a34-4b6c-9963-39037a8d7523' 2024-09-10T17:06:38.531 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:06:38.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCefOBm1FgNIBAAiduEpBAjkuraI2DkRrOOWQ== 2024-09-10T17:06:38.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCefOBm1FgNIBAAiduEpBAjkuraI2DkRrOOWQ=="}' 2024-09-10T17:06:38.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0ace607f-4a34-4b6c-9963-39037a8d7523 -i td/test-erasure-eio/4/new.json 2024-09-10T17:06:39.353 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:06:39.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:06:39.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCefOBm1FgNIBAAiduEpBAjkuraI2DkRrOOWQ== --osd-uuid 0ace607f-4a34-4b6c-9963-39037a8d7523 2024-09-10T17:06:39.394 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:39.393+0000 7f84bf317640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:39.396 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:39.394+0000 7f84bf317640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:39.397 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:39.395+0000 7f84bf317640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:39.397 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:39.396+0000 7f84bf317640 -1 bdev(0x556741183c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:06:39.397 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:39.396+0000 7f84bf317640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:06:46.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:06:46.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:06:46.112 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:06:46.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:06:46.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:06:46.648 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:06:46.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:06:46.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:06:46.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:06:46.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:06:46.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:06:46.675 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:46.673+0000 7f25abcdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:46.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:46.704+0000 7f25abcdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:46.706 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:46.705+0000 7f25abcdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:06:47.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:06:47.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:06:47.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:06:47.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:47.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:47.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:48.298 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:48.296+0000 7f25abcdc640 -1 Falling back to public interface 2024-09-10T17:06:48.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:48.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:48.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:06:48.514 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:06:48.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:48.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:48.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:49.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:49.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:49.946 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:06:49.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:06:49.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:49.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:50.100 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:06:50.099+0000 7f25abcdc640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:06:50.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:51.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:51.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:51.470 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:06:51.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:06:51.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:51.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:52.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:53.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:53.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:53.206 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:06:53.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:06:53.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:53.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:53.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:54.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:54.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:54.682 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:06:54.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:06:54.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:54.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:55.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:56.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:56.108 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:06:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:06:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:56.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:56.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:57.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:57.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:57.553 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:06:57.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:06:57.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:57.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:57.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:06:58.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:06:58.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:06:58.994 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:06:58.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:06:58.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:06:58.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:06:59.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:00.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:00.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:00.390 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:07:00.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:07:00.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:00.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:00.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:01.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:01.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:01.788 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:07:01.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:07:01.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:01.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:02.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:03.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:03.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:03.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:07:03.302 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:07:03.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:03.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:04.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:05.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:05.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:05.553 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:07:05.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:07:05.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:05.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:06.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:07.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:07.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:07.407 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:07:07.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:07:07.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:07.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:07.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:08.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:08.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:08.974 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:07:08.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:07:08.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:08.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:09.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:10.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:10.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:10.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:07:10.835 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:07:10.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:10.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:11.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:12.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:12.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:12.671 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:07:12.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:07:12.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:12.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:13.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:14.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:14.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:14.296 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:07:14.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:07:14.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:14.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:07:14.895 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2938184966,v1:127.0.0.1:6835/2938184966] [v2:127.0.0.1:6836/2938184966,v1:127.0.0.1:6837/2938184966] exists,up 0ace607f-4a34-4b6c-9963-39037a8d7523 2024-09-10T17:07:14.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:07:14.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:07:14.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:07:14.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3898ac40-3810-42fd-9321-6dbb34fa30f3 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 3898ac40-3810-42fd-9321-6dbb34fa30f3' 2024-09-10T17:07:14.903 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 3898ac40-3810-42fd-9321-6dbb34fa30f3 2024-09-10T17:07:14.904 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:07:14.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCfOBmsd+mNhAAh9tXZOVyBx9K1JngENZJhw== 2024-09-10T17:07:14.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCfOBmsd+mNhAAh9tXZOVyBx9K1JngENZJhw=="}' 2024-09-10T17:07:14.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3898ac40-3810-42fd-9321-6dbb34fa30f3 -i td/test-erasure-eio/5/new.json 2024-09-10T17:07:15.884 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:07:15.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-10T17:07:15.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCfOBmsd+mNhAAh9tXZOVyBx9K1JngENZJhw== --osd-uuid 3898ac40-3810-42fd-9321-6dbb34fa30f3 2024-09-10T17:07:15.926 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:15.925+0000 7fad99c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:15.959 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:15.958+0000 7fad99c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:15.960 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:15.959+0000 7fad99c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:15.960 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:15.959+0000 7fad99c0f640 -1 bdev(0x562324651c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-10T17:07:15.960 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:15.959+0000 7fad99c0f640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-10T17:07:22.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-10T17:07:22.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:07:22.718 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T17:07:22.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T17:07:22.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:07:23.285 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T17:07:23.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T17:07:23.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:07:23.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:07:23.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:07:23.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:07:23.313 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:23.311+0000 7f5b21f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:23.327 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:23.326+0000 7f5b21f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:23.328 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:23.327+0000 7f5b21f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T17:07:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:07:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T17:07:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:07:23.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:07:23.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:23.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:07:23.751 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:07:23.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:23.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:24.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:24.924 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:24.923+0000 7f5b21f27640 -1 Falling back to public interface 2024-09-10T17:07:25.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:25.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:25.169 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:07:25.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:07:25.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:25.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:25.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:26.498 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:26.497+0000 7f5b21f27640 -1 osd.5 0 log_to_monitors true 2024-09-10T17:07:26.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:26.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:26.556 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:07:26.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:07:26.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:26.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:27.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:28.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:28.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:28.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:07:28.179 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:07:28.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:28.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:28.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:29.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:29.643 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:07:29.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:29.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:07:29.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:29.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:30.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:31.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:31.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:31.102 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:07:31.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:07:31.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:31.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:31.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:32.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:32.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:32.552 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:07:32.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:07:32.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:32.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:33.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:34.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:34.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:34.098 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:07:34.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:07:34.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:34.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:34.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:35.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:07:35.628 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:07:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:36.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:37.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:37.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:37.034 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:07:37.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:07:37.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:37.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:37.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:38.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:38.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:38.544 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:07:38.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:07:38.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:38.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:38.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:39.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:39.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:39.990 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:07:39.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:07:39.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:39.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:40.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:41.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:41.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:41.782 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:07:41.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:07:41.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:41.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:42.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:43.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:43.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:43.668 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:07:43.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:07:43.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:43.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:44.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:45.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:45.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:45.330 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:07:45.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:07:45.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:45.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:46.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:47.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:47.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:47.107 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:07:47.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:07:47.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:47.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:48.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:49.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:49.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:49.127 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:07:49.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:07:49.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:49.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:49.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:07:50.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:07:50.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:51.000 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:07:51.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:07:51.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:51.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:07:51.396 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 47 up_thru 49 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2468880465,v1:127.0.0.1:6843/2468880465] [v2:127.0.0.1:6844/2468880465,v1:127.0.0.1:6845/2468880465] exists,up 3898ac40-3810-42fd-9321-6dbb34fa30f3 2024-09-10T17:07:51.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:07:51.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:07:51.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:07:51.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:07:51.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:07:51.402 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:07:51.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=520f9db8-b4bf-4d29-925b-593a6c6a65e2 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 520f9db8-b4bf-4d29-925b-593a6c6a65e2 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 520f9db8-b4bf-4d29-925b-593a6c6a65e2' 2024-09-10T17:07:51.404 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:07:51.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnfOBmfU3dGBAAMBb6TllfRK7gCF3Us9zfmA== 2024-09-10T17:07:51.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnfOBmfU3dGBAAMBb6TllfRK7gCF3Us9zfmA=="}' 2024-09-10T17:07:51.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 520f9db8-b4bf-4d29-925b-593a6c6a65e2 -i td/test-erasure-eio/6/new.json 2024-09-10T17:07:51.965 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:07:51.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-10T17:07:51.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnfOBmfU3dGBAAMBb6TllfRK7gCF3Us9zfmA== --osd-uuid 520f9db8-b4bf-4d29-925b-593a6c6a65e2 2024-09-10T17:07:52.007 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:52.006+0000 7f9a72222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:52.009 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:52.008+0000 7f9a72222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:52.010 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:52.009+0000 7f9a72222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:52.011 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:52.009+0000 7f9a72222640 -1 bdev(0x555a1207fc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-10T17:07:52.011 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:52.010+0000 7f9a72222640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-10T17:07:58.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-10T17:07:58.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:07:58.630 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T17:07:58.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T17:07:58.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:07:59.184 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T17:07:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T17:07:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:07:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:07:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:07:59.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:07:59.212 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:59.210+0000 7fd33cf41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:59.223 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:59.222+0000 7fd33cf41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:59.224 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:07:59.223+0000 7fd33cf41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:07:59.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:07:59.639 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:07:59.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:07:59.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:00.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:00.330 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:00.329+0000 7fd33cf41640 -1 Falling back to public interface 2024-09-10T17:08:01.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:01.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:01.019 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:08:01.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:08:01.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:01.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:01.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:01.994 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:01.992+0000 7fd33cf41640 -1 osd.6 0 log_to_monitors true 2024-09-10T17:08:02.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:02.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:02.406 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:08:02.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:08:02.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:02.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:02.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:03.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:03.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:03.802 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:08:03.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:08:03.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:03.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:04.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:05.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:05.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:05.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:08:05.623 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:08:05.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:05.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:06.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:07.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:07.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:07.122 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:08:07.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:08:07.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:07.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:07.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:08.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:08.537 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:08:08.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:08.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:08:08.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:08.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:10.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:10.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:10.069 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:08:10.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:08:10.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:10.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:10.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:11.506 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:08:11.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:08:11.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:11.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:11.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:12.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:12.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:12.910 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:08:12.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:08:12.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:12.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:13.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:14.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:14.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:14.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:08:14.305 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:08:14.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:14.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:14.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:15.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:15.797 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:15.797 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:08:15.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:08:15.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:15.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:16.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:17.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:17.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:17.435 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:08:17.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:08:17.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:17.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:17.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:18.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:18.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:18.888 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:08:18.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:08:18.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:18.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:19.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:20.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:20.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:20.570 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:08:20.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:08:20.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:20.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:21.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:22.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:22.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:22.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:08:22.089 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:08:22.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:22.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:22.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:23.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:23.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:23.877 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:08:23.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:08:23.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:23.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:24.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:25.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:25.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:25.622 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:08:25.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:08:25.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:25.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:26.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:27.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:27.200 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:08:27.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:08:27.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:27.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:08:27.631 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 57 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3451818291,v1:127.0.0.1:6851/3451818291] [v2:127.0.0.1:6852/3451818291,v1:127.0.0.1:6853/3451818291] exists,up 520f9db8-b4bf-4d29-925b-593a6c6a65e2 2024-09-10T17:08:27.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:08:27.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:08:27.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:08:27.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:08:27.713 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:08:27.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:08:27.724 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:04:30.972+0000 7f1c5200f640 0 load: jerasure load: lrc 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:08:27.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:08:28.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:08:28.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:08:29.791 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:08:29.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:08:30.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:08:30.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:08:30.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:08:30.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:08:30.804 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:08:30.805 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:08:30.805 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:08:30.805 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:08:30.805 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:08:30.805 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:08:30.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:08:30.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:08:30.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:08:30.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:08:30.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:08:30.903 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:08:31.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.368 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:08:31.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803831 2024-09-10T17:08:31.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803831 2024-09-10T17:08:31.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831' 2024-09-10T17:08:31.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:08:31.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574891 2024-09-10T17:08:31.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574891 2024-09-10T17:08:31.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891' 2024-09-10T17:08:31.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:08:31.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411362 2024-09-10T17:08:31.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411362 2024-09-10T17:08:31.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-77309411362' 2024-09-10T17:08:31.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:08:31.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117018 2024-09-10T17:08:31.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117018 2024-09-10T17:08:31.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-77309411362 3-115964117018' 2024-09-10T17:08:31.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:08:31.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757275 2024-09-10T17:08:31.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757275 2024-09-10T17:08:31.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-77309411362 3-115964117018 4-163208757275' 2024-09-10T17:08:31.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:31.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:08:32.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462926 2024-09-10T17:08:32.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462926 2024-09-10T17:08:32.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-77309411362 3-115964117018 4-163208757275 5-201863462926' 2024-09-10T17:08:32.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:32.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:08:32.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-10T17:08:32.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-10T17:08:32.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-77309411362 3-115964117018 4-163208757275 5-201863462926 6-244813135879' 2024-09-10T17:08:32.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:32.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803831 2024-09-10T17:08:32.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:32.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:08:32.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803831 2024-09-10T17:08:32.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:32.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803831 2024-09-10T17:08:32.184 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803831 2024-09-10T17:08:32.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-10T17:08:32.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:08:32.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803831 2024-09-10T17:08:32.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:08:33.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:08:33.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:08:33.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-10T17:08:33.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:33.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574891 2024-09-10T17:08:33.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:33.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:08:33.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574891 2024-09-10T17:08:33.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:33.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574891 2024-09-10T17:08:33.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-09-10T17:08:33.996 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574891 2024-09-10T17:08:33.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:08:34.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574891 -lt 55834574891 2024-09-10T17:08:34.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:34.421 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411362 2024-09-10T17:08:34.421 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:34.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:08:34.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411362 2024-09-10T17:08:34.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:34.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411362 2024-09-10T17:08:34.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411362' 2024-09-10T17:08:34.423 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411362 2024-09-10T17:08:34.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:08:34.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411363 -lt 77309411362 2024-09-10T17:08:34.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:34.848 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117018 2024-09-10T17:08:34.848 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:34.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:08:34.849 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117018 2024-09-10T17:08:34.849 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:34.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117018 2024-09-10T17:08:34.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117018' 2024-09-10T17:08:34.852 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117018 2024-09-10T17:08:34.852 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:08:35.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117019 -lt 115964117018 2024-09-10T17:08:35.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:35.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757275 2024-09-10T17:08:35.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:35.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:08:35.235 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757275 2024-09-10T17:08:35.236 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:35.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757275 2024-09-10T17:08:35.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757275' 2024-09-10T17:08:35.237 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757275 2024-09-10T17:08:35.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:08:35.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757275 -lt 163208757275 2024-09-10T17:08:35.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:35.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462926 2024-09-10T17:08:35.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:35.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:08:35.652 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462926 2024-09-10T17:08:35.652 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:35.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462926 2024-09-10T17:08:35.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462926' 2024-09-10T17:08:35.654 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 201863462926 2024-09-10T17:08:35.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:08:36.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462926 -lt 201863462926 2024-09-10T17:08:36.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:36.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-244813135879 2024-09-10T17:08:36.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:36.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:08:36.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-244813135879 2024-09-10T17:08:36.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:36.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-10T17:08:36.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 244813135879' 2024-09-10T17:08:36.121 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 244813135879 2024-09-10T17:08:36.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:08:36.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135880 -lt 244813135879 2024-09-10T17:08:36.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:08:36.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:08:36.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:08:37.086 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:08:37.087 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:08:37.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:08:37.458 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:08:37.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:08:37.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:08:37.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:08:37.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:08:37.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:08:37.999 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-10T17:08:38.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2024-09-10T17:08:38.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2024-09-10T17:08:38.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:08:38.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T17:08:38.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-10T17:08:38.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-10T17:08:38.426 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:08:38.426 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:08:38.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:08:38.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:08:38.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:08:38.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:08:38.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:08:38.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:08:38.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-10T17:08:40.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-10T17:08:40.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:08:40.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:08:40.277 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:08:40.277 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:08:40.277 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:08:40.727 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2147483647 2024-09-10T17:08:40.727 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:08:40.727 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2147483647 5 1 0 6 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=5 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:08:40.728 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/5/type 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=5 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.5 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-10T17:08:40.730 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-10T17:08:40.731 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:40.731 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:40.731 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:40.731 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.5.asok 2024-09-10T17:08:40.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:08:40.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:08:40.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.5 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.5.asok 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:08:40.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-10T17:08:40.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-10T17:08:40.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:08:40.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:08:40.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:08:40.905 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:08:40.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:08:40.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:08:40.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:08:40.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:08:40.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:08:40.907 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:08:40.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:08:40.931 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:40.930+0000 7f55cd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:40.933 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:40.931+0000 7f55cd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:40.934 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:40.932+0000 7f55cd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:41.321 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:08:41.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:08:41.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:41.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:08:41.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:41.779 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:41.778+0000 7f55cd61e640 -1 Falling back to public interface 2024-09-10T17:08:42.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:42.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:42.756 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:08:42.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:08:42.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:42.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:08:43.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:08:43.852 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:43.850+0000 7f55cd61e640 -1 osd.3 65 log_to_monitors true 2024-09-10T17:08:44.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:08:44.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:08:44.166 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:08:44.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:08:44.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:08:44.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:08:44.656 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [27,65) [v2:127.0.0.1:6826/4105103436,v1:127.0.0.1:6827/4105103436] [v2:127.0.0.1:6828/4105103436,v1:127.0.0.1:6829/4105103436] exists,up 2ea8baa6-9832-4e43-bd27-5e579581e7e9 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:08:44.658 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:08:44.659 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:08:44.659 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:08:44.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:08:44.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:08:44.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:08:44.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:08:44.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:08:44.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:08:44.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:08:44.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:08:44.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:08:45.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:08:45.133 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:08:45.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803835 2024-09-10T17:08:45.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803835 2024-09-10T17:08:45.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835' 2024-09-10T17:08:45.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:08:45.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574895 2024-09-10T17:08:45.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574895 2024-09-10T17:08:45.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895' 2024-09-10T17:08:45.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:08:45.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411367 2024-09-10T17:08:45.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411367 2024-09-10T17:08:45.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-77309411367' 2024-09-10T17:08:45.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.576 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:08:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743427 2024-09-10T17:08:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743427 2024-09-10T17:08:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-77309411367 3-296352743427' 2024-09-10T17:08:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.690 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:08:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757280 2024-09-10T17:08:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757280 2024-09-10T17:08:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-77309411367 3-296352743427 4-163208757280' 2024-09-10T17:08:45.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:08:45.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462930 2024-09-10T17:08:45.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462930 2024-09-10T17:08:45.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-77309411367 3-296352743427 4-163208757280 5-201863462930' 2024-09-10T17:08:45.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:08:45.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:08:46.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135884 2024-09-10T17:08:46.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135884 2024-09-10T17:08:46.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-77309411367 3-296352743427 4-163208757280 5-201863462930 6-244813135884' 2024-09-10T17:08:46.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:46.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803835 2024-09-10T17:08:46.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:46.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:08:46.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803835 2024-09-10T17:08:46.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:46.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803835 2024-09-10T17:08:46.030 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803835 2024-09-10T17:08:46.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-10T17:08:46.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:08:46.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803833 -lt 25769803835 2024-09-10T17:08:46.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:08:47.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:08:47.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:08:47.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803836 -lt 25769803835 2024-09-10T17:08:47.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:47.879 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574895 2024-09-10T17:08:47.879 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:47.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:08:47.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574895 2024-09-10T17:08:47.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:47.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574895 2024-09-10T17:08:47.882 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574895 2024-09-10T17:08:47.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-10T17:08:47.882 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:08:48.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574896 -lt 55834574895 2024-09-10T17:08:48.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:48.297 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411367 2024-09-10T17:08:48.297 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:48.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:08:48.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411367 2024-09-10T17:08:48.299 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411367 2024-09-10T17:08:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411367' 2024-09-10T17:08:48.300 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411367 2024-09-10T17:08:48.300 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:08:48.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411367 -lt 77309411367 2024-09-10T17:08:48.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:48.707 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-296352743427 2024-09-10T17:08:48.707 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:48.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:08:48.708 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-296352743427 2024-09-10T17:08:48.708 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:48.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743427 2024-09-10T17:08:48.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 296352743427' 2024-09-10T17:08:48.709 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 296352743427 2024-09-10T17:08:48.709 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:08:49.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743427 -lt 296352743427 2024-09-10T17:08:49.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:49.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757280 2024-09-10T17:08:49.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:49.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:08:49.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757280 2024-09-10T17:08:49.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:49.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757280 2024-09-10T17:08:49.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757280' 2024-09-10T17:08:49.136 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757280 2024-09-10T17:08:49.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:08:49.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757280 -lt 163208757280 2024-09-10T17:08:49.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:49.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-201863462930 2024-09-10T17:08:49.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:49.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:08:49.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-201863462930 2024-09-10T17:08:49.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:49.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462930 2024-09-10T17:08:49.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 201863462930' 2024-09-10T17:08:49.562 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 201863462930 2024-09-10T17:08:49.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:08:49.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462931 -lt 201863462930 2024-09-10T17:08:49.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-244813135884 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-244813135884 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135884 2024-09-10T17:08:49.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 244813135884' 2024-09-10T17:08:49.991 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 244813135884 2024-09-10T17:08:49.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:08:50.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135885 -lt 244813135884 2024-09-10T17:08:50.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:08:50.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:08:50.413 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:08:50.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:08:51.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:08:51.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:08:51.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:08:51.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:08:51.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-10T17:08:51.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:08:51.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:08:51.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:08:51.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:08:51.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:08:52.409 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:08:52.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:08:53.345 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:08:53.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:08:53.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:08:53.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:08:53.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:08:53.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:08:53.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:08:53.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:08:53.469 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:08:53.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:08:53.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:08:53.470 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:08:53.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:08:53.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:08:53.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:08:53.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:08:53.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:08:53.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:08:53.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:08:53.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:08:53.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:53.506 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:53.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:08:53.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:08:53.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:08:53.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:08:53.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:08:53.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:08:53.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:08:53.510 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:08:53.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:08:53.511 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:08:53.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:08:53.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:08:53.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:08:53.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:08:53.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:08:53.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:08:53.513 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:08:53.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:08:53.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:08:53.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:08:53.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:08:53.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:53.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:53.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:08:53.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:08:53.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:08:53.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:08:53.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:08:53.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:53.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:53.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:08:53.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:08:53.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:08:53.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:08:53.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:53.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:08:54.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:08:54.137 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:08:54.138 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:54.138 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:54.138 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:54.139 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:08:54.139 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:08:54.139 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:08:54.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:08:54.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:08:54.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:08:54.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:08:54.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:08:54.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:08:54.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:08:54.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:55.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:08:55.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:08:55.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:08:55.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:08:55.460 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:08:55.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:08:56.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:08:56.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:08:56.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:08:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:08:56.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:56.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:08:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:08:56.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:08:56.543 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:08:56.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:08:56.553 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:08:53.968+0000 7f71b61c3c80 0 load: jerasure load: lrc 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:439: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:08:56.558 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:08:56.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:08:56.560 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:08:56.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ca04de2-eebe-441c-8186-9f8e3c99de5b 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5ca04de2-eebe-441c-8186-9f8e3c99de5b' 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 5ca04de2-eebe-441c-8186-9f8e3c99de5b 2024-09-10T17:08:56.562 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:08:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAofeBmnB1cIhAAQSfjPCTGFddPLpbBBaA+gg== 2024-09-10T17:08:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAofeBmnB1cIhAAQSfjPCTGFddPLpbBBaA+gg=="}' 2024-09-10T17:08:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ca04de2-eebe-441c-8186-9f8e3c99de5b -i td/test-erasure-eio/0/new.json 2024-09-10T17:08:56.828 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:08:56.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:08:56.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAofeBmnB1cIhAAQSfjPCTGFddPLpbBBaA+gg== --osd-uuid 5ca04de2-eebe-441c-8186-9f8e3c99de5b 2024-09-10T17:08:56.868 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:56.867+0000 7f8212d29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:56.897 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:56.896+0000 7f8212d29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:56.898 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:56.897+0000 7f8212d29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:08:56.899 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:56.897+0000 7f8212d29640 -1 bdev(0x5578f0475c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:08:56.899 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:08:56.898+0000 7f8212d29640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:09:02.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:09:02.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:09:02.886 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:09:02.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:09:02.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:09:03.416 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:09:03.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:09:03.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:09:03.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:09:03.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:09:03.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:09:03.444 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:03.442+0000 7f001040f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:03.465 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:03.463+0000 7f001040f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:03.466 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:03.465+0000 7f001040f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:03.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:09:03.880 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:09:03.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:03.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:04.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:04.827 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:04.826+0000 7f001040f640 -1 Falling back to public interface 2024-09-10T17:09:05.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:05.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:05.289 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:09:05.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:09:05.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:05.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:05.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:06.709 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:06.708+0000 7f001040f640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:09:06.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:06.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:06.715 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:09:06.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:09:06.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:06.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:07.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:08.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:08.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:08.328 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:09:08.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:09:08.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:08.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:08.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:09.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:09.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:09.736 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:09:09.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:09:09.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:09.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:10.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:11.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:11.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:11.133 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:09:11.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:09:11.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:11.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:11.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:12.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:12.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:12.618 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:09:12.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:09:12.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:12.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:13.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:14.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:14.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:14.015 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:09:14.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:09:14.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:14.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:14.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:15.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:15.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:15.455 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:09:15.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:09:15.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:15.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:15.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:16.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:16.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:16.855 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:09:16.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:09:16.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:16.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:17.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:18.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:18.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:18.281 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:09:18.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:09:18.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:18.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:18.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:19.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:19.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:19.913 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:09:19.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:09:19.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:19.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:20.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:21.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:21.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:21.304 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:09:21.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:09:21.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:21.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:21.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:22.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:22.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:22.694 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:09:22.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:09:22.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:22.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:23.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:24.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:24.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:24.403 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:09:24.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:09:24.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:24.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:25.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:26.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:26.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:26.123 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:09:26.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:09:26.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:26.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:26.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:27.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:27.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:27.972 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:09:27.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:09:27.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:27.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:28.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:29.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:29.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:29.769 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:09:29.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:09:29.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:29.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:09:30.205 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3876323400,v1:127.0.0.1:6803/3876323400] [v2:127.0.0.1:6804/3876323400,v1:127.0.0.1:6805/3876323400] exists,up 5ca04de2-eebe-441c-8186-9f8e3c99de5b 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:09:30.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:09:30.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:09:30.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3' 2024-09-10T17:09:30.212 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3 2024-09-10T17:09:30.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:09:30.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKfeBmFoFxDRAA1ij0EHtMkA4qd5Yf1zjiBA== 2024-09-10T17:09:30.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKfeBmFoFxDRAA1ij0EHtMkA4qd5Yf1zjiBA=="}' 2024-09-10T17:09:30.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3 -i td/test-erasure-eio/1/new.json 2024-09-10T17:09:30.704 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:09:30.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:09:30.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKfeBmFoFxDRAA1ij0EHtMkA4qd5Yf1zjiBA== --osd-uuid 368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3 2024-09-10T17:09:30.745 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:30.743+0000 7fc8c500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:30.756 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:30.753+0000 7fc8c500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:30.756 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:30.754+0000 7fc8c500f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:30.756 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:30.754+0000 7fc8c500f640 -1 bdev(0x55ed38879c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:09:30.756 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:30.754+0000 7fc8c500f640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:09:36.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:09:36.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:09:36.986 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:09:36.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:09:36.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:09:37.524 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:09:37.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:09:37.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:09:37.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:09:37.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:09:37.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:09:37.551 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:37.550+0000 7ffa94754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:37.553 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:37.552+0000 7ffa94754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:37.554 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:37.553+0000 7ffa94754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:09:37.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:09:37.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:09:37.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:09:37.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:09:37.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:09:37.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:37.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:09:37.943 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:09:37.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:37.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:38.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:38.920 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:38.919+0000 7ffa94754640 -1 Falling back to public interface 2024-09-10T17:09:39.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:39.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:39.378 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:09:39.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:09:39.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:39.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:39.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:40.515 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:09:40.513+0000 7ffa94754640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:09:40.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:40.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:40.765 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:09:40.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:09:40.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:40.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:41.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:42.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:42.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:42.336 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:09:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:09:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:42.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:43.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:43.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:43.758 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:09:43.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:09:43.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:43.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:44.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:45.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:45.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:45.150 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:09:45.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:09:45.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:45.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:45.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:46.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:46.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:46.765 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:09:46.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:09:46.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:46.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:47.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:48.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:48.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:48.162 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:09:48.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:09:48.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:48.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:48.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:49.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:49.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:49.702 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:09:49.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:09:49.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:49.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:50.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:51.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:51.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:51.141 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:09:51.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:09:51.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:51.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:51.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:52.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:52.793 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:09:52.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:52.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:09:52.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:52.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:53.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:54.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:54.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:54.228 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:09:54.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:09:54.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:54.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:54.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:55.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:55.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:55.939 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:09:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:09:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:57.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:58.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:58.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:58.270 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:09:58.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:09:58.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:58.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:09:58.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:09:59.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:09:59.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:09:59.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:09:59.808 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:09:59.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:09:59.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:10:00.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:01.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:01.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:01.802 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:10:01.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:10:01.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:01.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:10:02.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:03.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:03.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:03.376 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:10:03.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:10:03.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:03.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:10:03.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:04.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:04.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:04.916 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:10:04.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:10:04.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:04.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:10:05.431 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2646982078,v1:127.0.0.1:6811/2646982078] [v2:127.0.0.1:6812/2646982078,v1:127.0.0.1:6813/2646982078] exists,up 368e7fa8-9670-4fd2-bdfd-4f8b26bb33c3 2024-09-10T17:10:05.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:10:05.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:10:05.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:10:05.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:10:05.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:10:05.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ba9ad2b-f467-44a1-9129-cd3d744e0633 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5ba9ad2b-f467-44a1-9129-cd3d744e0633' 2024-09-10T17:10:05.441 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 5ba9ad2b-f467-44a1-9129-cd3d744e0633 2024-09-10T17:10:05.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:10:05.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtfeBm8tfyGhAA30xtNnlyBo6K7Knc1WszQg== 2024-09-10T17:10:05.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtfeBm8tfyGhAA30xtNnlyBo6K7Knc1WszQg=="}' 2024-09-10T17:10:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ba9ad2b-f467-44a1-9129-cd3d744e0633 -i td/test-erasure-eio/2/new.json 2024-09-10T17:10:05.933 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:10:05.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:10:05.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtfeBm8tfyGhAA30xtNnlyBo6K7Knc1WszQg== --osd-uuid 5ba9ad2b-f467-44a1-9129-cd3d744e0633 2024-09-10T17:10:05.974 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:05.972+0000 7f192a6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:05.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:05.982+0000 7f192a6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:05.984 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:05.983+0000 7f192a6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:05.983+0000 7f192a6e2640 -1 bdev(0x55d2585f1c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:10:05.985 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:05.984+0000 7f192a6e2640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:10:11.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:10:11.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:10:11.644 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:10:11.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:10:11.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:10:12.176 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:10:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:10:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:10:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:10:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:10:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:10:12.204 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:12.202+0000 7fa79ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:12.219 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:12.217+0000 7fa79ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:12.220 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:12.219+0000 7fa79ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:12.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:10:12.596 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:10:12.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:12.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:13.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:13.569 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:13.568+0000 7fa79ae1f640 -1 Falling back to public interface 2024-09-10T17:10:14.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:14.019 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:10:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:10:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:14.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:15.164 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:15.162+0000 7fa79ae1f640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:10:15.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:15.438 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:10:15.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:15.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:10:15.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:15.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:15.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:16.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:16.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:16.838 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:10:16.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:10:16.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:16.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:17.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:18.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:18.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:18.390 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:10:18.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:10:18.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:18.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:18.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:19.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:19.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:19.813 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:10:19.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:10:19.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:19.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:20.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:21.254 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:10:21.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:10:21.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:21.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:21.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:22.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:22.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:22.723 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:10:22.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:10:22.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:22.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:23.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:24.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:24.205 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:10:24.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:24.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:10:24.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:24.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:24.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:25.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:25.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:25.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:10:25.605 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:10:25.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:25.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:26.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:27.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:27.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:27.052 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:10:27.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:10:27.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:27.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:27.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:28.443 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:10:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:10:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:29.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:30.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:30.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:30.058 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:10:30.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:10:30.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:30.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:30.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:31.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:31.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:31.980 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:10:31.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:10:31.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:31.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:32.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:33.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:33.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:33.503 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:10:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:10:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:34.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:35.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:35.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:35.132 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:10:35.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:10:35.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:35.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:35.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:36.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:36.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:36.628 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:10:36.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:10:36.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:36.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:37.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:38.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:38.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:38.346 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:10:38.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:10:38.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:38.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:10:38.768 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/93072080,v1:127.0.0.1:6819/93072080] [v2:127.0.0.1:6820/93072080,v1:127.0.0.1:6821/93072080] exists,up 5ba9ad2b-f467-44a1-9129-cd3d744e0633 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:10:38.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:10:38.773 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:10:38.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a3065071-71e1-47a5-9296-e5a965fa7234 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a3065071-71e1-47a5-9296-e5a965fa7234' 2024-09-10T17:10:38.775 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 a3065071-71e1-47a5-9296-e5a965fa7234 2024-09-10T17:10:38.776 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:10:38.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOfeBmjCYGLxAAfvhQodzfi52PVQT1SmFeDg== 2024-09-10T17:10:38.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOfeBmjCYGLxAAfvhQodzfi52PVQT1SmFeDg=="}' 2024-09-10T17:10:38.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a3065071-71e1-47a5-9296-e5a965fa7234 -i td/test-erasure-eio/3/new.json 2024-09-10T17:10:39.635 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:10:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:10:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOfeBmjCYGLxAAfvhQodzfi52PVQT1SmFeDg== --osd-uuid a3065071-71e1-47a5-9296-e5a965fa7234 2024-09-10T17:10:39.676 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:39.674+0000 7f7e14811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:39.685 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:39.683+0000 7f7e14811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:39.686 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:39.685+0000 7f7e14811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:39.687 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:39.685+0000 7f7e14811640 -1 bdev(0x55aa67b4bc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:10:39.687 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:39.685+0000 7f7e14811640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:10:46.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:10:46.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:10:46.305 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:10:46.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:10:46.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:10:46.835 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:10:46.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:10:46.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:10:46.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:10:46.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:10:46.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:10:46.863 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:46.861+0000 7f910b42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:46.880 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:46.878+0000 7f910b42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:46.881 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:46.880+0000 7f910b42c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:10:47.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:10:47.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:10:47.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:10:47.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:10:47.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:10:47.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:47.285 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:10:47.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:10:47.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:47.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:47.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:47.986 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:47.985+0000 7f910b42c640 -1 Falling back to public interface 2024-09-10T17:10:48.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:48.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:48.746 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:10:48.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:10:48.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:48.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:49.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:49.791 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:10:49.789+0000 7f910b42c640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:10:50.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:50.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:50.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:10:50.215 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:10:50.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:50.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:50.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:51.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:51.779 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:10:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:10:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:52.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:53.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:53.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:53.253 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:10:53.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:10:53.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:53.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:53.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:54.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:54.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:54.744 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:10:54.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:10:54.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:54.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:55.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:56.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:56.135 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:10:56.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:10:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:56.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:56.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:57.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:57.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:57.528 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:10:57.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:10:57.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:57.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:57.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:10:58.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:10:58.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:10:58.962 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:10:58.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:10:58.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:10:58.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:10:59.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:00.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:00.386 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:11:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:11:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:00.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:00.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:01.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:01.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:01.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:11:01.970 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:11:01.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:01.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:02.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:03.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:03.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:03.389 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:11:03.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:11:03.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:03.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:03.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:04.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:04.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:04.862 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:11:04.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:11:04.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:04.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:05.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:06.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:06.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:06.399 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:11:06.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:11:06.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:06.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:07.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:08.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:08.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:08.131 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:11:08.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:11:08.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:08.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:08.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:09.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:09.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:09.900 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:11:09.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:11:09.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:09.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:11.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:12.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:12.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:12.102 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:11:12.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:11:12.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:12.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:12.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:13.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:13.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:13.875 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:11:13.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:11:13.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:13.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:11:14.365 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/208952978,v1:127.0.0.1:6827/208952978] [v2:127.0.0.1:6828/208952978,v1:127.0.0.1:6829/208952978] exists,up a3065071-71e1-47a5-9296-e5a965fa7234 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:11:14.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:11:14.371 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:11:14.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8dfe7bda-7cad-48ae-8ade-127d78e97e4c 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 8dfe7bda-7cad-48ae-8ade-127d78e97e4c' 2024-09-10T17:11:14.373 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 8dfe7bda-7cad-48ae-8ade-127d78e97e4c 2024-09-10T17:11:14.374 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:11:14.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCyfeBmquAFFxAAGvrQmbZLiP9bZAx4SQlKAQ== 2024-09-10T17:11:14.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCyfeBmquAFFxAAGvrQmbZLiP9bZAx4SQlKAQ=="}' 2024-09-10T17:11:14.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8dfe7bda-7cad-48ae-8ade-127d78e97e4c -i td/test-erasure-eio/4/new.json 2024-09-10T17:11:14.895 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:11:14.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:11:14.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyfeBmquAFFxAAGvrQmbZLiP9bZAx4SQlKAQ== --osd-uuid 8dfe7bda-7cad-48ae-8ade-127d78e97e4c 2024-09-10T17:11:14.936 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:14.934+0000 7f328aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:14.937 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:14.936+0000 7f328aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:14.938 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:14.937+0000 7f328aa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:14.939 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:14.938+0000 7f328aa13640 -1 bdev(0x55e832a9fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:11:14.939 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:14.938+0000 7f328aa13640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:11:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:11:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:11:21.301 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:11:21.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:11:21.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:11:21.840 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:11:21.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:11:21.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:11:21.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:11:21.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:11:21.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:11:21.867 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:21.865+0000 7fe9701d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:21.869 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:21.867+0000 7fe9701d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:21.870 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:21.868+0000 7fe9701d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:11:22.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:22.260 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:11:22.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:11:22.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:22.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:22.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:22.715 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:22.714+0000 7fe9701d9640 -1 Falling back to public interface 2024-09-10T17:11:23.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:23.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:23.643 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:11:23.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:11:23.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:23.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:24.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:24.447 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:24.445+0000 7fe9701d9640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:11:25.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:25.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:25.051 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:11:25.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:11:25.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:25.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:25.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:26.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:26.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:26.728 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:11:26.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:11:26.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:26.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:27.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:28.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:28.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:28.463 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:11:28.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:11:28.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:28.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:28.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:29.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:29.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:29.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:11:29.895 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:11:29.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:29.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:30.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:31.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:31.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:31.343 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:11:31.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:11:31.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:31.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:31.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:32.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:32.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:32.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:11:32.883 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:11:32.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:32.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:33.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:34.384 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:11:34.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:34.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:34.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:11:34.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:34.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:34.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:35.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:35.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:35.800 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:11:35.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:11:35.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:35.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:36.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:37.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:37.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:37.357 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:11:37.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:11:37.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:37.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:37.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:38.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:38.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:38.906 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:11:38.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:11:38.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:38.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:39.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:40.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:40.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:40.757 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:11:40.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:11:40.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:40.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:41.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:42.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:42.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:42.685 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:11:42.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:11:42.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:42.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:43.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:44.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:44.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:44.455 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:11:44.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:11:44.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:45.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:46.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:46.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:46.257 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:11:46.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:11:46.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:46.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:47.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:48.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:48.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:48.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:11:48.002 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:11:48.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:48.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:48.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:49.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:11:49.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:49.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:11:49.830 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:11:49.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:49.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:11:50.447 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4111520797,v1:127.0.0.1:6835/4111520797] [v2:127.0.0.1:6836/4111520797,v1:127.0.0.1:6837/4111520797] exists,up 8dfe7bda-7cad-48ae-8ade-127d78e97e4c 2024-09-10T17:11:50.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:11:50.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:11:50.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:11:50.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:11:50.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:11:50.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=28cc1c4e-9198-458b-be1f-e68c0e7045d0 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 28cc1c4e-9198-458b-be1f-e68c0e7045d0' 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 28cc1c4e-9198-458b-be1f-e68c0e7045d0 2024-09-10T17:11:50.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:11:50.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWfeBmAlDiGxAAv4gL2h2sFvXms/8KaBigpA== 2024-09-10T17:11:50.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWfeBmAlDiGxAAv4gL2h2sFvXms/8KaBigpA=="}' 2024-09-10T17:11:50.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 28cc1c4e-9198-458b-be1f-e68c0e7045d0 -i td/test-erasure-eio/5/new.json 2024-09-10T17:11:51.183 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:11:51.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-10T17:11:51.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWfeBmAlDiGxAAv4gL2h2sFvXms/8KaBigpA== --osd-uuid 28cc1c4e-9198-458b-be1f-e68c0e7045d0 2024-09-10T17:11:51.224 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:51.222+0000 7f764ed25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:51.226 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:51.224+0000 7f764ed25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:51.227 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:51.225+0000 7f764ed25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:51.227 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:51.226+0000 7f764ed25640 -1 bdev(0x55f12b9ffc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-10T17:11:51.227 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:51.226+0000 7f764ed25640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-10T17:11:57.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-10T17:11:57.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:11:57.718 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T17:11:57.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T17:11:57.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:11:58.259 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T17:11:58.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T17:11:58.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:11:58.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:11:58.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:11:58.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:11:58.287 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:58.285+0000 7f559f3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:58.309 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:58.307+0000 7f559f3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:58.310 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:58.308+0000 7f559f3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:11:58.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:11:58.724 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:11:58.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:11:58.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:11:59.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:11:59.403 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:11:59.402+0000 7f559f3bb640 -1 Falling back to public interface 2024-09-10T17:12:00.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:00.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:00.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:12:00.114 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:12:00.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:00.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:00.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:01.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:01.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:01.535 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:12:01.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:12:01.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:01.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:01.605 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:01.603+0000 7f559f3bb640 -1 osd.5 0 log_to_monitors true 2024-09-10T17:12:02.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:03.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:03.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:03.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:12:03.015 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:12:03.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:03.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:03.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:04.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:04.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:04.522 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:12:04.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:12:04.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:04.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:05.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:06.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:06.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:06.106 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:12:06.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:12:06.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:06.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:06.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:07.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:07.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:07.515 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:12:07.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:12:07.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:07.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:08.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:09.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:09.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:09.048 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:12:09.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:12:09.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:09.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:09.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:10.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:10.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:10.574 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:12:10.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:12:10.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:10.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:10.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:11.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:11.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:11.991 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:12:11.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:12:11.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:11.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:12.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:13.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:13.541 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:12:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:12:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:14.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:15.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:15.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:15.171 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:12:15.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:12:15.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:15.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:15.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:16.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:16.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:16.758 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:12:16.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:12:16.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:16.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:17.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:18.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:18.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:18.542 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:12:18.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:12:18.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:18.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:19.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:20.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:20.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:20.355 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:12:20.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:12:20.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:20.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:21.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:22.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:22.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:22.157 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:12:22.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:12:22.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:22.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:22.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:23.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:23.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:23.999 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:12:23.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:12:23.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:24.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:24.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:25.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:25.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:25.873 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:12:25.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:12:25.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:25.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:12:26.440 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 48 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3440873546,v1:127.0.0.1:6843/3440873546] [v2:127.0.0.1:6844/3440873546,v1:127.0.0.1:6845/3440873546] exists,up 28cc1c4e-9198-458b-be1f-e68c0e7045d0 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:12:26.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:12:26.445 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:12:26.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2566ba70-e4fc-4303-8acf-19c8c36043a4 2024-09-10T17:12:26.447 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 2566ba70-e4fc-4303-8acf-19c8c36043a4 2024-09-10T17:12:26.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 2566ba70-e4fc-4303-8acf-19c8c36043a4' 2024-09-10T17:12:26.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:12:26.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6feBmYyl+GxAAjDWgp6eGvYi2BlJZeJYd9A== 2024-09-10T17:12:26.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6feBmYyl+GxAAjDWgp6eGvYi2BlJZeJYd9A=="}' 2024-09-10T17:12:26.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2566ba70-e4fc-4303-8acf-19c8c36043a4 -i td/test-erasure-eio/6/new.json 2024-09-10T17:12:27.076 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:12:27.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-10T17:12:27.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6feBmYyl+GxAAjDWgp6eGvYi2BlJZeJYd9A== --osd-uuid 2566ba70-e4fc-4303-8acf-19c8c36043a4 2024-09-10T17:12:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:27.116+0000 7fe1c5a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:27.206 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:27.205+0000 7fe1c5a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:27.207 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:27.206+0000 7fe1c5a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:27.207 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:27.206+0000 7fe1c5a15640 -1 bdev(0x55976e407c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-10T17:12:27.208 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:27.206+0000 7fe1c5a15640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-10T17:12:33.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-10T17:12:33.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:12:33.461 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T17:12:33.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T17:12:33.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:12:33.994 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T17:12:33.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T17:12:33.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:12:33.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:12:33.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:12:33.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:12:34.022 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:34.020+0000 7fad3a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:34.045 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:34.043+0000 7fad3a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:34.046 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:34.044+0000 7fad3a80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:12:34.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:34.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:12:34.460 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:12:34.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:34.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:34.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:34.902 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:34.901+0000 7fad3a80d640 -1 Falling back to public interface 2024-09-10T17:12:35.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:35.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:35.870 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:12:35.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:12:35.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:35.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:36.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:36.657 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:12:36.655+0000 7fad3a80d640 -1 osd.6 0 log_to_monitors true 2024-09-10T17:12:37.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:37.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:37.262 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:12:37.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:12:37.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:37.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:37.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:38.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:38.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:38.931 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:12:38.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:12:38.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:38.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:39.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:40.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:40.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:40.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:12:40.639 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:12:40.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:40.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:41.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:42.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:42.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:42.063 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:12:42.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:12:42.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:42.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:42.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:43.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:43.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:43.471 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:12:43.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:12:43.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:43.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:44.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:45.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:45.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:45.009 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:12:45.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:12:45.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:45.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:45.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:46.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:46.557 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:12:46.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:46.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:12:46.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:46.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:46.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:47.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:47.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:47.949 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:12:47.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:12:47.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:47.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:48.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:49.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:49.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:49.473 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:12:49.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:12:49.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:49.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:50.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:51.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:51.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:51.594 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:12:51.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:12:51.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:51.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:52.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:53.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:53.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:53.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:12:53.413 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:12:53.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:53.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:54.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:55.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:55.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:55.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:12:55.249 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:12:55.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:55.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:55.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:56.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:56.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:56.717 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:12:56.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:12:56.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:56.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:57.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:12:58.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:12:58.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:12:58.488 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:12:58.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:12:58.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:12:58.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:12:59.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:00.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:00.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:00.651 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:13:00.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:13:00.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:00.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:13:01.117 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 58 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1011262069,v1:127.0.0.1:6851/1011262069] [v2:127.0.0.1:6852/1011262069,v1:127.0.0.1:6853/1011262069] exists,up 2566ba70-e4fc-4303-8acf-19c8c36043a4 2024-09-10T17:13:01.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:13:01.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:13:01.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:13:01.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-10T17:13:01.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:13:01.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:13:01.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:13:01.125 INFO:tasks.workunit.client.0.smithi086.stdout:add osd7 749f40de-c302-49d7-8eea-69464a4c69d9 2024-09-10T17:13:01.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=749f40de-c302-49d7-8eea-69464a4c69d9 2024-09-10T17:13:01.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 749f40de-c302-49d7-8eea-69464a4c69d9' 2024-09-10T17:13:01.126 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:13:01.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdfuBmqGA9CBAAa1TV3964xaWq9ePtgSzkvg== 2024-09-10T17:13:01.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdfuBmqGA9CBAAa1TV3964xaWq9ePtgSzkvg=="}' 2024-09-10T17:13:01.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 749f40de-c302-49d7-8eea-69464a4c69d9 -i td/test-erasure-eio/7/new.json 2024-09-10T17:13:02.168 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:13:02.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-10T17:13:02.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdfuBmqGA9CBAAa1TV3964xaWq9ePtgSzkvg== --osd-uuid 749f40de-c302-49d7-8eea-69464a4c69d9 2024-09-10T17:13:02.210 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:02.209+0000 7f43a9102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:02.218 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:02.216+0000 7f43a9102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:02.219 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:02.217+0000 7f43a9102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:02.220 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:02.218+0000 7f43a9102640 -1 bdev(0x55a570199c00 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2024-09-10T17:13:02.220 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:02.218+0000 7f43a9102640 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-10T17:13:09.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-10T17:13:09.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:13:09.431 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd7 key to auth repository 2024-09-10T17:13:09.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-10T17:13:09.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:13:09.965 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.7 2024-09-10T17:13:09.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-10T17:13:09.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:13:09.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:13:09.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:13:09.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:13:09.993 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:09.992+0000 7f7fb0c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:10.015 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:10.013+0000 7f7fb0c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:10.016 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:10.014+0000 7f7fb0c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:10.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:13:10.422 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:13:10.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:10.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:10.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:11.384 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:11.383+0000 7f7fb0c0d640 -1 Falling back to public interface 2024-09-10T17:13:11.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:11.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:11.839 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:13:11.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:13:11.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:11.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:12.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:12.989+0000 7f7fb0c0d640 -1 osd.7 0 log_to_monitors true 2024-09-10T17:13:13.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:13.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:13.276 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:13:13.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:13:13.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:13.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:13.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:14.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:14.908 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:13:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:13:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:15.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:16.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:16.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:16.424 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:13:16.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:13:16.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:16.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:16.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:17.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:17.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:17.961 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:13:17.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:13:17.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:17.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:18.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:19.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:19.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:19.426 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:13:19.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:13:19.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:19.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:19.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:20.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:20.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:20.966 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:13:20.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:13:20.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:20.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:21.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:22.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:22.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:22.417 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:13:22.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:13:22.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:22.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:22.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:23.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:23.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:23.933 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:13:23.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:13:23.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:23.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:24.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:25.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:25.342 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:13:25.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:13:25.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:25.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:25.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:26.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:26.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:26.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:13:26.751 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:13:26.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:26.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:27.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:28.485 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:13:28.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:13:28.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:28.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:29.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:30.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:30.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:30.281 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:13:30.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:13:30.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:30.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:31.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:32.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:32.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:32.150 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:13:32.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:13:32.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:32.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:32.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:33.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:33.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:33.862 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:13:33.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:13:33.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:33.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:34.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:35.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:35.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:35.783 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:13:35.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:13:35.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:35.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:36.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:37.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:37.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:37.553 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:13:37.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:13:37.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:37.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-10T17:13:38.112 INFO:tasks.workunit.client.0.smithi086.stdout:osd.7 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/668246064,v1:127.0.0.1:6859/668246064] [v2:127.0.0.1:6860/668246064,v1:127.0.0.1:6861/668246064] exists,up 749f40de-c302-49d7-8eea-69464a4c69d9 2024-09-10T17:13:38.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:13:38.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:13:38.118 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:13:38.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ccf3c3df-0f98-45ae-8daf-dd684350575b 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 ccf3c3df-0f98-45ae-8daf-dd684350575b' 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stdout:add osd8 ccf3c3df-0f98-45ae-8daf-dd684350575b 2024-09-10T17:13:38.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:13:38.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCfuBmM+bOBxAAshGtCaFNpWh0bz3a0HHf2w== 2024-09-10T17:13:38.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCfuBmM+bOBxAAshGtCaFNpWh0bz3a0HHf2w=="}' 2024-09-10T17:13:38.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ccf3c3df-0f98-45ae-8daf-dd684350575b -i td/test-erasure-eio/8/new.json 2024-09-10T17:13:39.266 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:13:39.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-10T17:13:39.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCfuBmM+bOBxAAshGtCaFNpWh0bz3a0HHf2w== --osd-uuid ccf3c3df-0f98-45ae-8daf-dd684350575b 2024-09-10T17:13:39.309 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:39.307+0000 7fea6421c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:39.332 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:39.331+0000 7fea6421c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:39.334 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:39.332+0000 7fea6421c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:39.334 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:39.332+0000 7fea6421c640 -1 bdev(0x5625f05d5c00 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2024-09-10T17:13:39.334 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:39.332+0000 7fea6421c640 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-10T17:13:46.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-10T17:13:46.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:13:46.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-10T17:13:46.421 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd8 key to auth repository 2024-09-10T17:13:46.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:13:46.951 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.8 2024-09-10T17:13:46.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-10T17:13:46.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:13:46.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:13:46.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:13:46.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:13:46.979 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:46.978+0000 7f66b480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:47.004 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:47.003+0000 7f66b480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:47.005 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:47.004+0000 7f66b480d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:13:47.394 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:13:47.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:47.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:13:47.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:47.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:47.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:48.370 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:48.369+0000 7f66b480d640 -1 Falling back to public interface 2024-09-10T17:13:48.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:48.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:48.822 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:13:48.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:13:48.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:48.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:49.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:50.208 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:13:50.207+0000 7f66b480d640 -1 osd.8 0 log_to_monitors true 2024-09-10T17:13:50.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:50.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:50.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:13:50.248 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:13:50.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:50.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:50.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:51.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:51.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:51.958 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:13:51.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:13:51.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:51.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:53.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:53.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:53.403 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:13:53.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:13:53.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:53.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:53.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:54.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:54.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:54.857 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:13:54.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:13:54.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:54.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:55.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:56.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:56.322 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:13:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:13:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:56.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:56.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:57.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:57.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:57.737 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:13:57.738 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:13:57.738 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:57.738 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:58.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:13:59.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:13:59.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:13:59.212 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:13:59.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:13:59.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:13:59.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:13:59.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:00.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:00.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:00.721 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:14:00.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:14:00.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:00.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:01.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:02.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:02.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:02.116 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:14:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:14:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:02.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:02.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:03.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:03.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:03.568 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:14:03.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:14:03.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:03.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:04.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:05.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:05.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:05.132 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:14:05.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:14:05.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:05.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:05.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:06.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:06.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:06.845 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:14:06.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:14:06.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:06.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:07.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:08.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:08.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:08.557 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:14:08.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:14:08.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:08.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:09.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:10.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:10.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:10.460 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:14:10.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:14:10.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:10.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:11.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:12.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:12.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:12.373 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:14:12.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:14:12.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:12.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:12.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:14.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:14.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:14.001 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:14:14.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:14:14.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:14.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:15.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:16.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:16.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:16.240 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:14:16.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:14:16.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:16.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-10T17:14:16.797 INFO:tasks.workunit.client.0.smithi086.stdout:osd.8 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3711721547,v1:127.0.0.1:6867/3711721547] [v2:127.0.0.1:6868/3711721547,v1:127.0.0.1:6869/3711721547] exists,up ccf3c3df-0f98-45ae-8daf-dd684350575b 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:16.799 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:14:16.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:14:16.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:14:16.878 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:14:16.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:14:16.892 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:09:04.830+0000 7f001040f640 0 load: jerasure load: lrc 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-10T17:14:16.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:14:16.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-10T17:14:18.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:14:18.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:14:19.007 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:14:19.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:14:20.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:14:20.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:14:20.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:14:20.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:14:20.020 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:14:20.021 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:14:20.021 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:14:20.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:14:20.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:14:20.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:14:20.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:14:20.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:14:20.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:14:20.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:14:20.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:14:20.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:14:20.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:14:20.502 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:8' 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:20.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:14:20.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803843 2024-09-10T17:14:20.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803843 2024-09-10T17:14:20.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843' 2024-09-10T17:14:20.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:20.645 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:14:20.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574902 2024-09-10T17:14:20.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574902 2024-09-10T17:14:20.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902' 2024-09-10T17:14:20.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:20.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:14:21.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345972 2024-09-10T17:14:21.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345972 2024-09-10T17:14:21.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972' 2024-09-10T17:14:21.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.063 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:14:21.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084330 2024-09-10T17:14:21.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084330 2024-09-10T17:14:21.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330' 2024-09-10T17:14:21.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:14:21.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757286 2024-09-10T17:14:21.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757286 2024-09-10T17:14:21.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330 4-163208757286' 2024-09-10T17:14:21.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.331 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:14:21.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430237 2024-09-10T17:14:21.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430237 2024-09-10T17:14:21.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330 4-163208757286 5-206158430237' 2024-09-10T17:14:21.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.444 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:14:21.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103190 2024-09-10T17:14:21.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103190 2024-09-10T17:14:21.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330 4-163208757286 5-206158430237 6-249108103190' 2024-09-10T17:14:21.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.555 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T17:14:21.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743439 2024-09-10T17:14:21.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743439 2024-09-10T17:14:21.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330 4-163208757286 5-206158430237 6-249108103190 7-296352743439' 2024-09-10T17:14:21.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:21.668 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T17:14:21.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350983 2024-09-10T17:14:21.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350983 2024-09-10T17:14:21.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803843 1-55834574902 2-85899345972 3-120259084330 4-163208757286 5-206158430237 6-249108103190 7-296352743439 8-347892350983' 2024-09-10T17:14:21.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:21.779 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803843 2024-09-10T17:14:21.779 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:21.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:14:21.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803843 2024-09-10T17:14:21.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:21.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803843 2024-09-10T17:14:21.781 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803843 2024-09-10T17:14:21.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-09-10T17:14:21.782 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:14:22.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803843 -lt 25769803843 2024-09-10T17:14:22.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:22.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574902 2024-09-10T17:14:22.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:22.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:14:22.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574902 2024-09-10T17:14:22.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:22.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574902 2024-09-10T17:14:22.189 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574902 2024-09-10T17:14:22.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574902' 2024-09-10T17:14:22.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:14:22.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574902 -lt 55834574902 2024-09-10T17:14:22.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:22.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345972 2024-09-10T17:14:22.614 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:22.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:14:22.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345972 2024-09-10T17:14:22.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:22.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345972 2024-09-10T17:14:22.617 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 85899345972 2024-09-10T17:14:22.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345972' 2024-09-10T17:14:22.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:14:23.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345972 -lt 85899345972 2024-09-10T17:14:23.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:23.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084330 2024-09-10T17:14:23.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:23.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:14:23.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084330 2024-09-10T17:14:23.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:23.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084330 2024-09-10T17:14:23.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084330' 2024-09-10T17:14:23.036 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084330 2024-09-10T17:14:23.036 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:14:23.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084330 -lt 120259084330 2024-09-10T17:14:23.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:23.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757286 2024-09-10T17:14:23.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:23.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:14:23.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757286 2024-09-10T17:14:23.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:23.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757286 2024-09-10T17:14:23.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757286' 2024-09-10T17:14:23.434 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757286 2024-09-10T17:14:23.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:14:23.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757286 -lt 163208757286 2024-09-10T17:14:23.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:23.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-206158430237 2024-09-10T17:14:23.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:23.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:14:23.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-206158430237 2024-09-10T17:14:23.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:23.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430237 2024-09-10T17:14:23.847 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 206158430237 2024-09-10T17:14:23.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 206158430237' 2024-09-10T17:14:23.848 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:14:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430237 -lt 206158430237 2024-09-10T17:14:24.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:24.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-249108103190 2024-09-10T17:14:24.280 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:24.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:14:24.281 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-249108103190 2024-09-10T17:14:24.281 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:24.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103190 2024-09-10T17:14:24.282 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 249108103190 2024-09-10T17:14:24.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 249108103190' 2024-09-10T17:14:24.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:14:24.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103190 -lt 249108103190 2024-09-10T17:14:24.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:24.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-296352743439 2024-09-10T17:14:24.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:24.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-10T17:14:24.699 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-296352743439 2024-09-10T17:14:24.699 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:24.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743439 2024-09-10T17:14:24.700 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.7 seq 296352743439 2024-09-10T17:14:24.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 296352743439' 2024-09-10T17:14:24.701 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-10T17:14:25.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743439 -lt 296352743439 2024-09-10T17:14:25.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:25.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-347892350983 2024-09-10T17:14:25.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:25.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T17:14:25.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-347892350983 2024-09-10T17:14:25.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:25.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350983 2024-09-10T17:14:25.118 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 347892350983 2024-09-10T17:14:25.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 347892350983' 2024-09-10T17:14:25.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T17:14:25.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350983 -lt 347892350983 2024-09-10T17:14:25.577 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:14:25.577 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:14:25.577 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:14:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:14:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:14:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:14:26.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:14:26.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:14:26.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:14:26.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:14:27.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:446: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:14:27.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:14:27.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:14:27.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-10T17:14:27.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:447: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:14:27.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:14:27.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:14:27.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:14:27.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:14:27.224 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:14:27.224 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:27.651 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:14:27.652 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:14:27.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:14:27.655 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:14:27.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:14:27.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:14:27.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:14:27.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:14:27.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:14:27.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-10T17:14:27.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=3 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:14:27.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:14:27.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:14:27.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:14:27.830 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:14:27.830 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:14:28.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:14:28.261 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:14:28.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/0/type 2024-09-10T17:14:28.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:14:28.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-10T17:14:28.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:14:28.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-10T17:14:28.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:14:28.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:14:28.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:14:28.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:14:28.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:14:28.350 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-10T17:14:28.350 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:14:28.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:14:28.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=4 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:14:28.439 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-10T17:14:28.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:14:28.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=6 2024-09-10T17:14:28.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:14:28.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/6/type 2024-09-10T17:14:28.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:14:28.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-10T17:14:28.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=6 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.6 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.6.asok 2024-09-10T17:14:28.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:14:28.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:14:28.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.6 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:28.952 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:28.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.6.asok 2024-09-10T17:14:28.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:14:28.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-10T17:14:29.038 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-10T17:14:29.038 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:14:29.038 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:14:29.038 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:14:29.039 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:29.427 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:7' 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:14:29.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:14:29.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:14:29.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:14:29.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:14:29.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-10T17:14:30.299 INFO:tasks.workunit.client.0.smithi086.stderr:osd.7 is already down. 2024-09-10T17:14:30.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-10T17:14:31.047 INFO:tasks.workunit.client.0.smithi086.stderr:osd.7 is already out. 2024-09-10T17:14:31.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-10T17:14:31.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:14:31.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:14:31.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:14:31.059 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:14:31.059 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:14:31.059 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:14:31.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:14:31.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:14:31.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:14:31.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:14:31.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:14:31.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:14:31.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:14:31.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:14:31.148 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:14:31.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:14:31.623 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:14:31.623 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:14:31.624 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:14:31.624 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:14:31.624 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:14:31.624 INFO:tasks.workunit.client.0.smithi086.stderr:6 2024-09-10T17:14:31.624 INFO:tasks.workunit.client.0.smithi086.stderr:7 2024-09-10T17:14:31.625 INFO:tasks.workunit.client.0.smithi086.stderr:8' 2024-09-10T17:14:31.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:14:31.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:31.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:14:31.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803847 2024-09-10T17:14:31.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803847 2024-09-10T17:14:31.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847' 2024-09-10T17:14:31.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:31.775 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:14:31.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574906 2024-09-10T17:14:31.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574906 2024-09-10T17:14:31.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906' 2024-09-10T17:14:31.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:31.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:14:32.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345976 2024-09-10T17:14:32.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345976 2024-09-10T17:14:32.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976' 2024-09-10T17:14:32.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.098 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:14:32.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084334 2024-09-10T17:14:32.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084334 2024-09-10T17:14:32.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976 3-120259084334' 2024-09-10T17:14:32.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.449 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:14:32.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757290 2024-09-10T17:14:32.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757290 2024-09-10T17:14:32.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976 3-120259084334 4-163208757290' 2024-09-10T17:14:32.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.562 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:14:32.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430241 2024-09-10T17:14:32.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430241 2024-09-10T17:14:32.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976 3-120259084334 4-163208757290 5-206158430241' 2024-09-10T17:14:32.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:14:32.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103194 2024-09-10T17:14:32.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103194 2024-09-10T17:14:32.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976 3-120259084334 4-163208757290 5-206158430241 6-249108103194' 2024-09-10T17:14:32.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.789 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-10T17:14:32.869 INFO:tasks.workunit.client.0.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-10T17:14:32.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T17:14:32.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T17:14:32.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T17:14:32.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:14:32.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350987 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350987 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803847 1-55834574906 2-85899345976 3-120259084334 4-163208757290 5-206158430241 6-249108103194 8-347892350987' 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803847 2024-09-10T17:14:33.017 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:33.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:14:33.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803847 2024-09-10T17:14:33.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:33.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803847 2024-09-10T17:14:33.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-09-10T17:14:33.020 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803847 2024-09-10T17:14:33.020 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:14:33.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803847 -lt 25769803847 2024-09-10T17:14:33.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:33.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574906 2024-09-10T17:14:33.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:33.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:14:33.425 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574906 2024-09-10T17:14:33.425 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:33.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574906 2024-09-10T17:14:33.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574906' 2024-09-10T17:14:33.427 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574906 2024-09-10T17:14:33.427 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:14:33.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574906 -lt 55834574906 2024-09-10T17:14:33.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:33.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345976 2024-09-10T17:14:33.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:33.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:14:33.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345976 2024-09-10T17:14:33.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:33.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345976 2024-09-10T17:14:33.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345976' 2024-09-10T17:14:33.889 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 85899345976 2024-09-10T17:14:33.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:14:34.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345976 -lt 85899345976 2024-09-10T17:14:34.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:34.330 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084334 2024-09-10T17:14:34.330 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:34.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:14:34.331 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084334 2024-09-10T17:14:34.331 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:34.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084334 2024-09-10T17:14:34.332 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084334 2024-09-10T17:14:34.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084334' 2024-09-10T17:14:34.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:14:34.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084334 -lt 120259084334 2024-09-10T17:14:34.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:34.758 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757290 2024-09-10T17:14:34.758 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:34.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:14:34.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757290 2024-09-10T17:14:34.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:34.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757290 2024-09-10T17:14:34.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757290' 2024-09-10T17:14:34.760 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757290 2024-09-10T17:14:34.761 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:14:35.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757290 -lt 163208757290 2024-09-10T17:14:35.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:35.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-206158430241 2024-09-10T17:14:35.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:35.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:14:35.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-206158430241 2024-09-10T17:14:35.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:35.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430241 2024-09-10T17:14:35.186 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 206158430241 2024-09-10T17:14:35.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 206158430241' 2024-09-10T17:14:35.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:14:35.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430241 -lt 206158430241 2024-09-10T17:14:35.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:35.576 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-249108103194 2024-09-10T17:14:35.576 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:35.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:14:35.577 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-249108103194 2024-09-10T17:14:35.577 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:35.578 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 249108103194 2024-09-10T17:14:35.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103194 2024-09-10T17:14:35.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 249108103194' 2024-09-10T17:14:35.579 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:14:36.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103194 -lt 249108103194 2024-09-10T17:14:36.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:14:36.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-347892350987 2024-09-10T17:14:36.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:14:36.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-10T17:14:36.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-347892350987 2024-09-10T17:14:36.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:14:36.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350987 2024-09-10T17:14:36.013 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.8 seq 347892350987 2024-09-10T17:14:36.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 347892350987' 2024-09-10T17:14:36.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-10T17:14:36.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350987 -lt 347892350987 2024-09-10T17:14:36.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:14:36.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:14:36.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:14:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:14:37.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:14:37.350 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:14:37.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:14:37.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:14:37.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:14:37.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:14:37.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:14:37.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-10T17:14:37.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:14:37.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:14:37.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:14:37.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:14:37.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:14:38.402 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:14:38.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:14:39.340 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:14:39.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:14:39.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:14:39.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:14:39.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:14:39.466 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:14:39.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:14:39.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:14:39.467 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:14:39.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:14:39.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:14:39.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:14:39.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:14:39.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:14:39.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:14:39.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:14:39.513 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:14:39.513 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:39.513 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:39.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:14:39.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:14:39.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:14:39.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:14:39.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:14:39.517 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:14:39.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:14:39.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:14:39.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:14:39.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:14:39.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:14:39.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:14:39.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:14:39.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:14:39.521 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:14:39.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:14:39.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:14:39.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:14:39.523 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:14:39.523 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:39.523 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:39.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:14:39.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:14:39.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:14:39.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:14:39.525 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:14:39.525 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:39.526 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:39.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:14:39.527 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:14:39.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:14:40.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:14:40.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:14:40.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:14:40.003 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:14:40.004 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:40.004 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:40.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:14:40.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:14:40.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:14:40.206 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:40.207 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:14:40.208 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:14:40.208 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:14:40.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:40.286 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:14:40.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:14:40.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:14:40.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:14:40.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:14:41.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:14:41.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:14:41.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:14:41.211 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:14:41.211 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:41.212 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:41.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:14:41.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:14:41.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:14:41.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:14:41.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:14:41.529 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:14:41.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:14:42.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:14:42.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:14:42.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:14:42.615 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:14:42.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:14:42.624 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:14:40.035+0000 7f4a5da12c80 0 load: jerasure load: lrc 2024-09-10T17:14:42.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-10T17:14:42.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-10T17:14:42.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-10T17:14:42.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:476: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-10T17:14:42.626 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:14:42.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:14:42.632 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:14:42.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:14:42.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4c758561-49e0-4b42-8ba6-0f4180386958 2024-09-10T17:14:42.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4c758561-49e0-4b42-8ba6-0f4180386958' 2024-09-10T17:14:42.635 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 4c758561-49e0-4b42-8ba6-0f4180386958 2024-09-10T17:14:42.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:14:42.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCfuBm3kbDJhAAAuXMeIiYj/Nvm8CTQcje2g== 2024-09-10T17:14:42.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCfuBm3kbDJhAAAuXMeIiYj/Nvm8CTQcje2g=="}' 2024-09-10T17:14:42.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4c758561-49e0-4b42-8ba6-0f4180386958 -i td/test-erasure-eio/0/new.json 2024-09-10T17:14:42.914 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:14:42.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:14:42.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCfuBm3kbDJhAAAuXMeIiYj/Nvm8CTQcje2g== --osd-uuid 4c758561-49e0-4b42-8ba6-0f4180386958 2024-09-10T17:14:42.958 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:42.956+0000 7fe6646da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:42.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:42.981+0000 7fe6646da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:42.984 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:42.982+0000 7fe6646da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:42.984 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:42.983+0000 7fe6646da640 -1 bdev(0x55c3f1be7c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:14:42.984 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:42.983+0000 7fe6646da640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:14:49.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:14:49.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:14:49.096 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:14:49.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:14:49.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:14:49.638 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:14:49.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:14:49.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:14:49.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:14:49.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:14:49.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:14:49.670 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:49.668+0000 7fb77ee0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:49.681 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:49.677+0000 7fb77ee0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:49.681 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:49.678+0000 7fb77ee0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:14:50.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:14:50.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:14:50.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:14:50.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:14:50.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:14:50.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:50.594 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:14:50.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:14:50.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:50.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:50.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:51.039 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:51.038+0000 7fb77ee0b640 -1 Falling back to public interface 2024-09-10T17:14:51.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:14:51.520 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:14:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:51.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:51.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:52.864 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:14:52.863+0000 7fb77ee0b640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:14:52.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:52.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:52.919 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:14:52.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:14:52.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:52.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:53.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:54.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:54.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:54.503 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:14:54.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:14:54.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:54.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:54.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:55.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:55.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:55.940 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:14:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:14:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:55.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:56.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:57.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:57.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:57.345 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:14:57.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:14:57.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:57.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:57.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:14:58.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:14:58.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:14:58.759 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:14:58.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:14:58.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:14:58.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:14:59.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:00.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:00.182 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:15:00.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:00.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:15:00.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:00.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:00.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:01.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:01.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:01.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:15:01.638 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:15:01.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:01.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:02.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:03.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:03.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:03.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:15:03.068 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:15:03.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:03.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:03.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:04.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:04.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:04.576 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:15:04.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:15:04.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:04.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:05.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:06.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:06.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:06.435 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:15:06.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:15:06.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:06.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:06.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:07.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:07.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:07.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:15:07.970 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:15:07.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:07.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:09.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:10.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:10.018 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:15:10.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:10.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:15:10.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:10.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:10.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:11.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:11.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:11.645 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:15:11.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:15:11.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:11.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:12.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:13.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:13.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:13.602 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:15:13.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:15:13.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:13.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:14.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:15.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:15.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:15.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:15:15.326 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:15:15.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:15.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:15:15.794 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1114174984,v1:127.0.0.1:6803/1114174984] [v2:127.0.0.1:6804/1114174984,v1:127.0.0.1:6805/1114174984] exists,up 4c758561-49e0-4b42-8ba6-0f4180386958 2024-09-10T17:15:15.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:15:15.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:15:15.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:15:15.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:15:15.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:15:15.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:15:15.801 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:15:15.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b4257aad-1875-40ed-b405-355a0f899c12 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b4257aad-1875-40ed-b405-355a0f899c12' 2024-09-10T17:15:15.803 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 b4257aad-1875-40ed-b405-355a0f899c12 2024-09-10T17:15:15.804 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:15:15.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjfuBmUyrPMBAAcNL5uV39bArVqL1Qtcf17A== 2024-09-10T17:15:15.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjfuBmUyrPMBAAcNL5uV39bArVqL1Qtcf17A=="}' 2024-09-10T17:15:15.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b4257aad-1875-40ed-b405-355a0f899c12 -i td/test-erasure-eio/1/new.json 2024-09-10T17:15:16.297 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:15:16.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:15:16.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjfuBmUyrPMBAAcNL5uV39bArVqL1Qtcf17A== --osd-uuid b4257aad-1875-40ed-b405-355a0f899c12 2024-09-10T17:15:16.339 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:16.337+0000 7fa86841a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:16.340 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:16.339+0000 7fa86841a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:16.341 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:16.340+0000 7fa86841a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:16.342 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:16.341+0000 7fa86841a640 -1 bdev(0x55f398173c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:15:16.342 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:16.341+0000 7fa86841a640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:15:22.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:15:22.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:15:22.725 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:15:22.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:15:22.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:15:23.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:15:23.267 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:15:23.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:15:23.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:15:23.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:15:23.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:15:23.299 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:23.297+0000 7f04c75c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:23.317 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:23.314+0000 7f04c75c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:23.317 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:23.316+0000 7f04c75c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:23.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:15:23.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:15:23.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:23.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:24.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:25.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:25.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:25.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:15:25.099 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:15:25.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:25.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:25.165 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:25.164+0000 7f04c75c5640 -1 Falling back to public interface 2024-09-10T17:15:25.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:26.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:26.559 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:15:26.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:26.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:15:26.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:26.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:26.737 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:26.735+0000 7f04c75c5640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:15:26.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:27.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:27.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:27.958 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:15:27.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:15:27.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:27.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:28.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:29.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:29.400 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:15:29.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:29.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:15:29.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:29.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:29.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:30.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:30.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:30.891 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:15:30.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:15:30.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:30.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:31.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:32.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:32.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:32.309 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:15:32.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:15:32.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:32.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:32.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:33.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:33.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:33.713 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:15:33.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:15:33.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:33.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:34.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:35.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:35.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:35.113 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:15:35.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:15:35.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:35.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:35.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:36.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:36.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:36.627 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:15:36.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:15:36.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:36.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:37.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:38.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:38.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:38.057 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:15:38.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:15:38.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:38.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:38.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:39.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:39.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:39.496 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:15:39.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:15:39.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:39.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:40.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:41.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:41.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:41.501 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:15:41.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:15:41.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:41.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:42.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:43.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:43.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:43.285 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:15:43.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:15:43.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:43.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:44.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:45.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:45.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:45.130 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:15:45.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:15:45.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:45.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:45.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:46.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:46.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:46.865 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:15:46.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:15:46.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:46.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:47.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:48.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:48.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:48.632 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:15:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:15:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:49.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:50.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:15:50.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:50.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:15:50.425 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:15:50.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:50.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:15:50.826 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3395253562,v1:127.0.0.1:6811/3395253562] [v2:127.0.0.1:6812/3395253562,v1:127.0.0.1:6813/3395253562] exists,up b4257aad-1875-40ed-b405-355a0f899c12 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:15:50.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:15:50.831 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:15:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=39a6ecd7-2c9f-4384-9037-ae2c0aa21555 2024-09-10T17:15:50.833 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 39a6ecd7-2c9f-4384-9037-ae2c0aa21555 2024-09-10T17:15:50.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 39a6ecd7-2c9f-4384-9037-ae2c0aa21555' 2024-09-10T17:15:50.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:15:50.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDGfuBmH0CqMhAAfrzcNqtxndHaIOmMvxet0Q== 2024-09-10T17:15:50.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDGfuBmH0CqMhAAfrzcNqtxndHaIOmMvxet0Q=="}' 2024-09-10T17:15:50.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 39a6ecd7-2c9f-4384-9037-ae2c0aa21555 -i td/test-erasure-eio/2/new.json 2024-09-10T17:15:51.447 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:15:51.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:15:51.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGfuBmH0CqMhAAfrzcNqtxndHaIOmMvxet0Q== --osd-uuid 39a6ecd7-2c9f-4384-9037-ae2c0aa21555 2024-09-10T17:15:51.493 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:51.491+0000 7fc48b4d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:51.494 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:51.493+0000 7fc48b4d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:51.495 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:51.494+0000 7fc48b4d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:51.496 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:51.494+0000 7fc48b4d4640 -1 bdev(0x55a918b0bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:15:51.496 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:51.495+0000 7fc48b4d4640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:15:57.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:15:57.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:15:57.748 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:15:57.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:15:57.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:15:58.282 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:15:58.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:15:58.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:15:58.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:15:58.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:15:58.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:15:58.314 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:58.313+0000 7f501a6b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:58.342+0000 7f501a6b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:58.345 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:58.343+0000 7f501a6b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:15:58.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:15:58.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:15:58.709 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:15:58.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:15:58.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:15:59.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:15:59.708 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:15:59.707+0000 7f501a6b6640 -1 Falling back to public interface 2024-09-10T17:16:00.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:00.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:00.139 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:16:00.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:16:00.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:00.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:00.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:01.339 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:01.338+0000 7f501a6b6640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:16:01.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:01.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:01.579 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:16:01.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:16:01.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:01.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:02.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:03.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:03.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:03.112 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:16:03.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:16:03.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:03.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:03.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:04.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:04.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:04.559 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:16:04.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:16:04.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:04.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:04.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:05.972 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:16:05.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:16:05.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:05.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:06.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:07.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:07.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:07.395 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:16:07.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:16:07.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:07.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:07.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:08.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:08.928 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:16:08.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:08.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:16:08.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:08.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:09.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:10.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:10.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:10.371 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:16:10.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:16:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:10.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:11.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:11.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:11.771 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:16:11.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:16:11.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:11.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:12.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:13.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:13.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:13.240 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:16:13.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:16:13.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:13.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:14.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:15.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:15.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:15.099 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:16:15.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:16:15.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:15.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:15.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:16.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:16.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:16.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:16:16.779 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:16:16.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:16.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:17.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:18.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:18.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:18.547 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:16:18.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:16:18.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:18.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:19.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:20.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:20.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:20.251 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:16:20.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:16:20.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:20.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:21.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:22.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:22.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:22.037 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:16:22.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:16:22.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:22.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:22.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:23.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:23.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:23.530 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:16:23.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:16:23.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:23.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:24.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:25.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:25.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:25.056 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:16:25.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:16:25.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:25.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:16:25.577 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2061227270,v1:127.0.0.1:6819/2061227270] [v2:127.0.0.1:6820/2061227270,v1:127.0.0.1:6821/2061227270] exists,up 39a6ecd7-2c9f-4384-9037-ae2c0aa21555 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:16:25.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:16:25.582 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:16:25.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a825ff24-eebf-4ea7-8f66-18936a043485 2024-09-10T17:16:25.584 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 a825ff24-eebf-4ea7-8f66-18936a043485 2024-09-10T17:16:25.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a825ff24-eebf-4ea7-8f66-18936a043485' 2024-09-10T17:16:25.585 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:16:25.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpfuBmFYjSIxAAZHt5817XAZ5fP0yD7/jqHQ== 2024-09-10T17:16:25.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpfuBmFYjSIxAAZHt5817XAZ5fP0yD7/jqHQ=="}' 2024-09-10T17:16:25.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a825ff24-eebf-4ea7-8f66-18936a043485 -i td/test-erasure-eio/3/new.json 2024-09-10T17:16:26.086 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:16:26.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:16:26.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpfuBmFYjSIxAAZHt5817XAZ5fP0yD7/jqHQ== --osd-uuid a825ff24-eebf-4ea7-8f66-18936a043485 2024-09-10T17:16:26.131 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:26.130+0000 7f0aeab89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:26.133 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:26.131+0000 7f0aeab89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:26.134 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:26.132+0000 7f0aeab89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:26.134 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:26.133+0000 7f0aeab89640 -1 bdev(0x55bb9717fc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:16:26.134 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:26.133+0000 7f0aeab89640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:16:32.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:16:32.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:16:32.113 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:16:32.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:16:32.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:16:32.651 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:16:32.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:16:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:16:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:16:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:16:32.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:16:32.684 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:32.683+0000 7f22b8e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:32.692 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:32.691+0000 7f22b8e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:32.693 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:32.692+0000 7f22b8e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:16:33.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:16:33.099 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:16:33.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:33.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:33.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:34.038+0000 7f22b8e13640 -1 Falling back to public interface 2024-09-10T17:16:34.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:34.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:34.492 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:16:34.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:16:34.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:34.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:34.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:35.713 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:16:35.711+0000 7f22b8e13640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:16:35.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:35.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:35.927 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:16:35.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:16:35.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:35.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:36.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:37.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:37.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:37.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:16:37.452 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:16:37.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:37.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:38.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:39.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:39.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:39.024 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:16:39.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:16:39.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:39.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:39.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:40.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:40.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:40.574 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:16:40.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:16:40.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:40.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:40.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:41.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:41.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:41.992 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:16:41.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:16:41.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:41.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:42.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:43.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:43.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:43.529 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:16:43.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:16:43.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:43.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:44.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:45.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:45.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:45.050 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:16:45.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:16:45.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:45.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:45.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:46.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:46.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:46.704 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:16:46.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:16:46.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:46.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:47.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:48.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:48.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:48.123 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:16:48.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:16:48.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:48.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:48.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:49.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:49.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:49.872 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:16:49.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:16:49.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:49.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:50.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:51.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:51.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:51.312 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:16:51.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:16:51.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:51.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:51.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:52.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:52.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:52.770 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:16:52.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:16:52.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:52.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:53.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:54.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:54.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:54.788 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:16:54.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:16:54.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:54.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:55.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:56.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:56.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:56.750 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:16:56.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:16:56.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:56.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:57.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:16:58.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:16:58.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:16:58.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:16:58.608 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:16:58.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:16:58.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:16:59.279 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1458198954,v1:127.0.0.1:6827/1458198954] [v2:127.0.0.1:6828/1458198954,v1:127.0.0.1:6829/1458198954] exists,up a825ff24-eebf-4ea7-8f66-18936a043485 2024-09-10T17:16:59.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:16:59.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:16:59.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:16:59.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:16:59.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1ec59a84-8343-4c97-af2e-86dab48da668 2024-09-10T17:16:59.287 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 1ec59a84-8343-4c97-af2e-86dab48da668 2024-09-10T17:16:59.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 1ec59a84-8343-4c97-af2e-86dab48da668' 2024-09-10T17:16:59.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:16:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALf+Bmph4YEhAAys5FAB0DTtKnPp/ywk+k6g== 2024-09-10T17:16:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALf+Bmph4YEhAAys5FAB0DTtKnPp/ywk+k6g=="}' 2024-09-10T17:16:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1ec59a84-8343-4c97-af2e-86dab48da668 -i td/test-erasure-eio/4/new.json 2024-09-10T17:17:00.202 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:17:00.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:17:00.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALf+Bmph4YEhAAys5FAB0DTtKnPp/ywk+k6g== --osd-uuid 1ec59a84-8343-4c97-af2e-86dab48da668 2024-09-10T17:17:00.248 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:00.246+0000 7ff2ff883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:00.250 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:00.248+0000 7ff2ff883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:00.251 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:00.249+0000 7ff2ff883640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:00.251 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:00.250+0000 7ff2ff883640 -1 bdev(0x56006820fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:17:00.252 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:00.250+0000 7ff2ff883640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:17:07.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:17:07.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:17:07.201 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:17:07.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:17:07.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:17:07.740 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:17:07.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:17:07.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:17:07.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:17:07.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:17:07.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:17:07.772 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:07.771+0000 7f8f84eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:07.785 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:07.783+0000 7f8f84eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:07.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:07.784+0000 7f8f84eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:17:08.199 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:17:08.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:08.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:08.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:08.638 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:08.637+0000 7f8f84eda640 -1 Falling back to public interface 2024-09-10T17:17:09.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:09.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:09.626 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:17:09.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:17:09.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:09.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:10.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:10.729 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:10.727+0000 7f8f84eda640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:17:11.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:11.061 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:17:11.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:11.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:17:11.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:11.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:11.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:12.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:12.701 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:17:12.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:12.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:17:12.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:12.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:14.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:14.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:14.611 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:17:14.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:17:14.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:14.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:15.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:16.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:16.069 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:17:16.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:16.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:17:16.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:16.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:16.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:17.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:17.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:17.593 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:17:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:17:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:17.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:18.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:18.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:18.997 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:17:18.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:17:18.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:18.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:19.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:20.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:20.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:20.462 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:17:20.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:17:20.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:20.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:20.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:21.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:21.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:21.918 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:17:21.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:17:21.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:21.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:22.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:23.326 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:17:23.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:23.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:17:23.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:23.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:24.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:25.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:25.047 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:17:25.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:25.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:17:25.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:25.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:25.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:26.541 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:17:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:26.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:17:26.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:26.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:27.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:28.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:28.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:28.261 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:17:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:17:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:28.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:29.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:30.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:30.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:30.104 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:17:30.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:17:30.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:30.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:30.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:31.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:31.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:31.875 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:17:31.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:17:31.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:31.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:32.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:33.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:33.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:33.578 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:17:33.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:17:33.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:33.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:34.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:35.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:35.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:35.625 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:17:35.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:17:35.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:35.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:17:36.181 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2925668195,v1:127.0.0.1:6835/2925668195] [v2:127.0.0.1:6836/2925668195,v1:127.0.0.1:6837/2925668195] exists,up 1ec59a84-8343-4c97-af2e-86dab48da668 2024-09-10T17:17:36.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:17:36.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:17:36.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:17:36.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:17:36.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-10T17:17:36.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:17:36.187 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:17:36.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=31a74600-6377-488a-8961-0da148a2edc6 2024-09-10T17:17:36.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 31a74600-6377-488a-8961-0da148a2edc6' 2024-09-10T17:17:36.189 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 31a74600-6377-488a-8961-0da148a2edc6 2024-09-10T17:17:36.189 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:17:36.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwf+BmjoVIDBAA47gytRUB65PB6sfWvb8zqw== 2024-09-10T17:17:36.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwf+BmjoVIDBAA47gytRUB65PB6sfWvb8zqw=="}' 2024-09-10T17:17:36.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 31a74600-6377-488a-8961-0da148a2edc6 -i td/test-erasure-eio/5/new.json 2024-09-10T17:17:36.985 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:17:36.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-10T17:17:36.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwf+BmjoVIDBAA47gytRUB65PB6sfWvb8zqw== --osd-uuid 31a74600-6377-488a-8961-0da148a2edc6 2024-09-10T17:17:37.031 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:37.030+0000 7f9eae20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:37.033 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:37.031+0000 7f9eae20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:37.034 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:37.033+0000 7f9eae20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:37.035 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:37.033+0000 7f9eae20e640 -1 bdev(0x5620161c3c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-10T17:17:37.035 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:37.033+0000 7f9eae20e640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-10T17:17:43.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-10T17:17:43.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:17:43.479 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T17:17:43.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T17:17:43.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:17:44.021 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T17:17:44.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T17:17:44.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:17:44.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:17:44.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:17:44.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:17:44.053 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:44.051+0000 7fb43e102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:44.063 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:44.061+0000 7fb43e102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:44.064 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:44.062+0000 7fb43e102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:17:44.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:44.438 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:17:44.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:17:44.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:44.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:44.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:45.653 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:45.652+0000 7fb43e102640 -1 Falling back to public interface 2024-09-10T17:17:45.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:45.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:45.825 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:17:45.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:17:45.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:45.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:46.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:47.250 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:17:47.248+0000 7fb43e102640 -1 osd.5 0 log_to_monitors true 2024-09-10T17:17:47.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:47.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:47.255 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:17:47.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:17:47.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:47.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:47.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:48.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:48.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:48.786 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:17:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:17:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:49.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:50.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:50.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:50.334 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:17:50.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:17:50.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:50.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:50.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:51.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:51.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:51.961 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:17:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:17:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:52.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:53.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:53.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:53.446 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:17:53.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:17:53.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:53.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:53.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:54.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:54.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:54.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:17:54.924 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:17:54.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:54.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:55.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:56.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:56.430 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:17:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:17:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:56.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:56.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:57.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:57.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:57.826 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:17:57.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:17:57.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:57.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:58.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:17:59.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:17:59.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:17:59.300 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:17:59.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:17:59.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:17:59.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:17:59.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:00.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:00.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:00.814 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:18:00.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:18:00.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:00.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:01.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:02.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:02.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:02.427 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:18:02.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:18:02.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:02.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:02.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:03.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:03.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:03.930 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:18:03.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:18:03.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:03.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:04.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:05.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:05.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:05.749 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:18:05.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:18:05.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:05.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:06.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:07.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:07.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:07.813 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:18:07.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:18:07.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:07.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:08.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:09.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:09.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:09.865 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:18:09.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:18:09.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:09.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:10.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:11.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:11.754 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:18:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:18:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:11.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:18:12.171 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1270890991,v1:127.0.0.1:6843/1270890991] [v2:127.0.0.1:6844/1270890991,v1:127.0.0.1:6845/1270890991] exists,up 31a74600-6377-488a-8961-0da148a2edc6 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-10T17:18:12.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:18:12.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:18:12.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1dafb26a-74e1-4b2f-84d0-89def79968b8 2024-09-10T17:18:12.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 1dafb26a-74e1-4b2f-84d0-89def79968b8' 2024-09-10T17:18:12.179 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 1dafb26a-74e1-4b2f-84d0-89def79968b8 2024-09-10T17:18:12.179 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:18:12.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUf+Bmd4WjCxAAvpDQhOcdWk2XapZwQnbsLw== 2024-09-10T17:18:12.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUf+Bmd4WjCxAAvpDQhOcdWk2XapZwQnbsLw=="}' 2024-09-10T17:18:12.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1dafb26a-74e1-4b2f-84d0-89def79968b8 -i td/test-erasure-eio/6/new.json 2024-09-10T17:18:12.656 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:18:12.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-10T17:18:12.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUf+Bmd4WjCxAAvpDQhOcdWk2XapZwQnbsLw== --osd-uuid 1dafb26a-74e1-4b2f-84d0-89def79968b8 2024-09-10T17:18:12.702 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:12.700+0000 7fb4baa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:12.707 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:12.706+0000 7fb4baa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:12.708 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:12.707+0000 7fb4baa2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:12.709 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:12.707+0000 7fb4baa2c640 -1 bdev(0x55f5b39e9c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-10T17:18:12.709 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:12.707+0000 7fb4baa2c640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-10T17:18:20.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-10T17:18:20.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:18:20.162 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T17:18:20.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T17:18:20.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:18:20.697 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T17:18:20.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T17:18:20.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:18:20.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:18:20.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:18:20.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:18:20.730 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:20.728+0000 7f48a5e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:20.739 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:20.738+0000 7f48a5e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:20.740 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:20.739+0000 7f48a5e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:18:21.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:18:21.154 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:18:21.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:21.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:21.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:21.588 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:21.587+0000 7f48a5e1d640 -1 Falling back to public interface 2024-09-10T17:18:22.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:22.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:22.546 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:18:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:18:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:22.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:23.443 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:18:23.441+0000 7f48a5e1d640 -1 osd.6 0 log_to_monitors true 2024-09-10T17:18:23.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:23.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:23.945 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:18:23.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:18:23.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:23.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:24.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:25.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:25.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:25.398 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:18:25.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:18:25.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:25.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:26.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:27.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:27.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:27.168 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:18:27.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:18:27.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:27.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:27.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:28.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:28.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:28.611 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:18:28.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:18:28.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:28.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:29.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:30.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:30.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:30.029 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:18:30.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:18:30.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:30.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:30.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:31.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:31.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:31.454 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:18:31.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:18:31.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:31.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:31.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:32.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:32.901 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:18:32.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:32.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:18:32.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:32.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:33.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:34.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:34.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:34.334 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:18:34.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:18:34.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:34.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:34.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:35.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:35.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:35.763 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:18:35.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:18:35.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:35.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:36.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:37.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:37.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:37.283 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:18:37.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:18:37.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:37.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:37.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:38.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:38.850 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:18:38.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:18:38.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:38.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:39.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:40.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:40.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:40.895 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:18:40.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:18:40.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:40.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:41.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:42.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:42.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:42.733 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:18:42.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:18:42.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:42.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:43.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:44.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:44.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:44.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:18:44.558 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:18:44.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:44.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:45.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:46.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:46.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:46.599 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:18:46.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:18:46.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:46.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:47.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:18:48.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:18:48.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:18:48.159 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:18:48.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:18:48.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:18:48.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:18:48.760 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3968962254,v1:127.0.0.1:6851/3968962254] [v2:127.0.0.1:6852/3968962254,v1:127.0.0.1:6853/3968962254] exists,up 1dafb26a-74e1-4b2f-84d0-89def79968b8 2024-09-10T17:18:48.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:18:48.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:18:48.840 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:18:48.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:18:48.852 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:14:51.042+0000 7fb77ee0b640 0 load: jerasure load: lrc 2024-09-10T17:18:48.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:477: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:18:48.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:18:48.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:18:49.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:18:49.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:18:50.955 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:18:50.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:18:51.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:18:51.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:18:51.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:18:51.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:18:51.968 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:18:51.969 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:18:51.969 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:18:51.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:18:51.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:18:51.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:18:52.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:18:52.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:18:52.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:18:52.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:18:52.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:18:52.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:18:52.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:52.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:18:52.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803824 2024-09-10T17:18:52.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803824 2024-09-10T17:18:52.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824' 2024-09-10T17:18:52.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:52.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:18:52.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574890 2024-09-10T17:18:52.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574890 2024-09-10T17:18:52.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890' 2024-09-10T17:18:52.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:52.726 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:18:52.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378659 2024-09-10T17:18:52.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378659 2024-09-10T17:18:52.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-81604378659' 2024-09-10T17:18:52.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:52.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:18:52.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117023 2024-09-10T17:18:52.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117023 2024-09-10T17:18:52.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-81604378659 3-115964117023' 2024-09-10T17:18:52.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:52.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:18:53.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757269 2024-09-10T17:18:53.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757269 2024-09-10T17:18:53.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-81604378659 3-115964117023 4-163208757269' 2024-09-10T17:18:53.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:53.062 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:18:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397519 2024-09-10T17:18:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397519 2024-09-10T17:18:53.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-81604378659 3-115964117023 4-163208757269 5-210453397519' 2024-09-10T17:18:53.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:53.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:18:53.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070470 2024-09-10T17:18:53.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070470 2024-09-10T17:18:53.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-81604378659 3-115964117023 4-163208757269 5-210453397519 6-253403070470' 2024-09-10T17:18:53.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:53.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803824 2024-09-10T17:18:53.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:53.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:18:53.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803824 2024-09-10T17:18:53.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:53.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803824 2024-09-10T17:18:53.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-10T17:18:53.288 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803824 2024-09-10T17:18:53.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:18:53.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-10T17:18:53.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:53.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574890 2024-09-10T17:18:53.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:53.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:18:53.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574890 2024-09-10T17:18:53.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:53.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574890 2024-09-10T17:18:53.716 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574890 2024-09-10T17:18:53.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574890' 2024-09-10T17:18:53.716 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:18:54.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574890 -lt 55834574890 2024-09-10T17:18:54.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:54.132 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378659 2024-09-10T17:18:54.132 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:54.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:18:54.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378659 2024-09-10T17:18:54.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:54.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378659 2024-09-10T17:18:54.135 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378659 2024-09-10T17:18:54.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378659' 2024-09-10T17:18:54.135 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:18:54.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378659 -lt 81604378659 2024-09-10T17:18:54.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:54.623 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117023 2024-09-10T17:18:54.624 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:54.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:18:54.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117023 2024-09-10T17:18:54.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:54.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117023 2024-09-10T17:18:54.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117023' 2024-09-10T17:18:54.626 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117023 2024-09-10T17:18:54.626 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:18:55.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117023 -lt 115964117023 2024-09-10T17:18:55.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:55.093 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757269 2024-09-10T17:18:55.093 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:55.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:18:55.095 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757269 2024-09-10T17:18:55.095 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:55.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757269 2024-09-10T17:18:55.096 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757269 2024-09-10T17:18:55.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757269' 2024-09-10T17:18:55.096 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:18:55.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757269 -lt 163208757269 2024-09-10T17:18:55.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:55.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-210453397519 2024-09-10T17:18:55.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:55.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:18:55.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-210453397519 2024-09-10T17:18:55.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:55.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397519 2024-09-10T17:18:55.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 210453397519' 2024-09-10T17:18:55.479 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 210453397519 2024-09-10T17:18:55.480 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:18:55.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397519 -lt 210453397519 2024-09-10T17:18:55.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:18:55.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070470 2024-09-10T17:18:55.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:18:55.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:18:55.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070470 2024-09-10T17:18:55.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:18:55.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070470 2024-09-10T17:18:55.915 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 253403070470 2024-09-10T17:18:55.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070470' 2024-09-10T17:18:55.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:18:56.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070470 -lt 253403070470 2024-09-10T17:18:56.322 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:18:56.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:18:56.323 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:18:56.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:18:56.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:18:57.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:18:57.273 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:18:57.274 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:18:57.274 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:57.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:18:57.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:57.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:18:57.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:57.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:18:57.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:18:58.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:18:58.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-10T17:18:58.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-10T17:18:58.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:18:58.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:18:58.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:18:58.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-10T17:18:58.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-10T17:18:59.086 INFO:tasks.workunit.client.0.smithi086.stderr:osd.0 is already out. 2024-09-10T17:18:59.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-10T17:18:59.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:18:59.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:18:59.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:18:59.098 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:18:59.098 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:18:59.098 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:18:59.098 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:18:59.099 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:18:59.099 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:18:59.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:18:59.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:59.672 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:18:59.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-10T17:18:59.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-10T17:18:59.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-10T17:18:59.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:18:59.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:19:00.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574893 2024-09-10T17:19:00.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574893 2024-09-10T17:19:00.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893' 2024-09-10T17:19:00.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:19:00.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:19:00.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378663 2024-09-10T17:19:00.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378663 2024-09-10T17:19:00.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893 2-81604378663' 2024-09-10T17:19:00.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:19:00.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:19:00.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117027 2024-09-10T17:19:00.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117027 2024-09-10T17:19:00.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893 2-81604378663 3-115964117027' 2024-09-10T17:19:00.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:19:00.524 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:19:00.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757273 2024-09-10T17:19:00.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757273 2024-09-10T17:19:00.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893 2-81604378663 3-115964117027 4-163208757273' 2024-09-10T17:19:00.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:19:00.636 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:19:00.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397522 2024-09-10T17:19:00.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397522 2024-09-10T17:19:00.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893 2-81604378663 3-115964117027 4-163208757273 5-210453397522' 2024-09-10T17:19:00.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:19:00.876 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:19:00.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070473 2024-09-10T17:19:00.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070473 2024-09-10T17:19:00.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-55834574893 2-81604378663 3-115964117027 4-163208757273 5-210453397522 6-253403070473' 2024-09-10T17:19:00.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:00.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-10T17:19:00.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:00.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:19:00.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-10T17:19:00.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:00.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-10T17:19:00.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-10T17:19:00.992 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803827 2024-09-10T17:19:00.993 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:19:01.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 0 -lt 25769803827 2024-09-10T17:19:01.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:19:02.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:19:02.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:19:02.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803827 2024-09-10T17:19:02.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:02.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574893 2024-09-10T17:19:02.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:02.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:19:02.779 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574893 2024-09-10T17:19:02.779 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:02.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574893 2024-09-10T17:19:02.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574893' 2024-09-10T17:19:02.781 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574893 2024-09-10T17:19:02.781 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:19:03.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574894 -lt 55834574893 2024-09-10T17:19:03.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:03.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378663 2024-09-10T17:19:03.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:03.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:19:03.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378663 2024-09-10T17:19:03.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:03.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378663 2024-09-10T17:19:03.170 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378663 2024-09-10T17:19:03.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378663' 2024-09-10T17:19:03.171 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:19:03.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378663 -lt 81604378663 2024-09-10T17:19:03.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:03.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117027 2024-09-10T17:19:03.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:03.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:19:03.586 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117027 2024-09-10T17:19:03.586 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:03.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117027 2024-09-10T17:19:03.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117027' 2024-09-10T17:19:03.587 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117027 2024-09-10T17:19:03.587 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:19:03.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117027 -lt 115964117027 2024-09-10T17:19:03.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:04.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757273 2024-09-10T17:19:04.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:04.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:19:04.001 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757273 2024-09-10T17:19:04.001 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:04.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757273 2024-09-10T17:19:04.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757273' 2024-09-10T17:19:04.003 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757273 2024-09-10T17:19:04.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:19:04.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757273 -lt 163208757273 2024-09-10T17:19:04.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:04.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-210453397522 2024-09-10T17:19:04.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:04.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:19:04.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-210453397522 2024-09-10T17:19:04.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:04.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397522 2024-09-10T17:19:04.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 210453397522' 2024-09-10T17:19:04.432 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 210453397522 2024-09-10T17:19:04.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:19:04.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397523 -lt 210453397522 2024-09-10T17:19:04.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:19:04.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-253403070473 2024-09-10T17:19:04.810 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:19:04.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:19:04.812 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-253403070473 2024-09-10T17:19:04.812 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:19:04.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070473 2024-09-10T17:19:04.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 253403070473' 2024-09-10T17:19:04.813 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 253403070473 2024-09-10T17:19:04.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:19:05.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070474 -lt 253403070473 2024-09-10T17:19:05.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:19:05.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:19:05.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:19:05.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:19:05.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:19:06.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:19:06.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:19:06.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:19:06.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-10T17:19:06.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:19:06.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:19:06.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-10T17:19:06.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:19:06.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:19:06.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:19:06.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-10T17:19:06.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:19:06.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:19:06.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-10T17:19:06.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:19:06.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:19:06.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:19:06.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:19:06.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:19:07.398 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:19:07.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:19:08.221 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:19:08.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:19:08.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:19:08.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:19:08.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:19:08.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:19:08.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:19:08.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:19:08.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:19:08.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:19:08.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:19:08.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:19:08.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:19:08.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:19:08.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:19:08.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:19:08.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:19:08.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:19:08.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:19:08.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:19:08.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:19:08.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:19:08.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:19:08.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:19:08.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:19:08.383 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:08.383 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:08.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:19:08.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:19:08.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:19:08.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:19:08.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:19:08.386 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:19:08.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:19:08.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:19:08.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:19:08.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:19:08.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:19:08.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:19:08.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:19:08.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:19:08.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:19:08.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:19:08.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:19:08.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:19:08.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:19:08.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:08.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:08.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:19:08.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:19:08.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:19:08.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:19:08.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:19:08.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:08.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:08.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:19:08.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:19:08.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:19:08.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:08.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:19:09.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:19:09.074 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:19:09.075 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:19:09.154 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:19:09.155 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:19:09.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:19:09.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:10.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:10.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:19:10.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:19:10.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:19:10.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:19:10.406 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:19:10.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:19:11.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:19:11.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:19:11.489 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:19:11.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:19:11.497 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:19:08.914+0000 7fa438019c80 0 load: jerasure load: lrc 2024-09-10T17:19:11.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-10T17:19:11.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-10T17:19:11.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-10T17:19:11.499 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-10T17:19:11.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:19:11.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:19:11.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:19:11.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:19:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:19:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:19:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:11.508 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:19:11.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:19:11.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:19:11.510 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:19:11.510 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 3bddfc8f-ec9c-462d-bc75-6b5798f047af 2024-09-10T17:19:11.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3bddfc8f-ec9c-462d-bc75-6b5798f047af 2024-09-10T17:19:11.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3bddfc8f-ec9c-462d-bc75-6b5798f047af' 2024-09-10T17:19:11.511 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:19:11.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCPf+BmWlJPHxAAxwHhuQnMlXNqJsnSt2BhJA== 2024-09-10T17:19:11.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCPf+BmWlJPHxAAxwHhuQnMlXNqJsnSt2BhJA=="}' 2024-09-10T17:19:11.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3bddfc8f-ec9c-462d-bc75-6b5798f047af -i td/test-erasure-eio/0/new.json 2024-09-10T17:19:11.791 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:19:11.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:19:11.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPf+BmWlJPHxAAxwHhuQnMlXNqJsnSt2BhJA== --osd-uuid 3bddfc8f-ec9c-462d-bc75-6b5798f047af 2024-09-10T17:19:11.835 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:11.833+0000 7f16a15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:11.859 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:11.858+0000 7f16a15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:11.861 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:11.859+0000 7f16a15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:11.861 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:11.859+0000 7f16a15cc640 -1 bdev(0x563ae5df9c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:19:11.861 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:11.860+0000 7f16a15cc640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:19:17.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:19:17.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:19:17.790 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:19:17.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:19:17.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:19:18.335 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:19:18.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:19:18.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:19:18.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:19:18.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:19:18.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:19:18.367 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:18.366+0000 7f2955a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:18.390 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:18.388+0000 7f2955a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:18.391 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:18.389+0000 7f2955a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:18.804 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:19:18.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:19:18.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:18.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:19.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:19.763 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:19.762+0000 7f2955a0e640 -1 Falling back to public interface 2024-09-10T17:19:20.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:20.231 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:19:20.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:20.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:19:20.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:20.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:20.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:21.592 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:21.590+0000 7f2955a0e640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:19:21.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:21.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:21.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:19:21.657 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:19:21.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:21.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:22.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:23.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:23.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:23.229 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:19:23.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:19:23.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:23.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:23.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:24.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:24.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:24.651 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:19:24.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:19:24.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:24.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:25.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:26.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:26.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:26.080 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:19:26.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:19:26.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:26.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:26.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:27.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:27.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:27.511 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:19:27.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:19:27.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:27.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:27.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:28.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:28.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:28.915 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:19:28.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:19:28.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:28.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:29.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:30.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:30.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:30.359 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:19:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:19:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:30.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:30.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:19:31.788 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:19:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:32.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:33.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:33.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:33.185 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:19:33.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:19:33.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:33.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:34.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:35.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:35.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:35.224 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:19:35.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:19:35.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:35.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:36.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:37.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:37.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:37.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:19:37.118 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:19:37.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:37.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:37.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:38.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:38.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:38.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:19:38.962 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:19:38.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:38.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:39.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:40.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:40.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:40.763 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:19:40.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:19:40.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:40.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:41.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:42.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:42.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:42.895 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:19:42.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:19:42.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:42.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:43.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:44.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:44.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:44.680 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:19:44.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:19:44.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:44.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:19:45.161 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2658736266,v1:127.0.0.1:6803/2658736266] [v2:127.0.0.1:6804/2658736266,v1:127.0.0.1:6805/2658736266] exists,up 3bddfc8f-ec9c-462d-bc75-6b5798f047af 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:19:45.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:19:45.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:19:45.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da4597c2-e108-47d4-bd4b-e2dde41d3564 2024-09-10T17:19:45.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 da4597c2-e108-47d4-bd4b-e2dde41d3564' 2024-09-10T17:19:45.169 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 da4597c2-e108-47d4-bd4b-e2dde41d3564 2024-09-10T17:19:45.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:19:45.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxf+Bm5p8DCxAA2uCUYpjyR21kn3cMwBTmsA== 2024-09-10T17:19:45.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxf+Bm5p8DCxAA2uCUYpjyR21kn3cMwBTmsA=="}' 2024-09-10T17:19:45.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da4597c2-e108-47d4-bd4b-e2dde41d3564 -i td/test-erasure-eio/1/new.json 2024-09-10T17:19:45.632 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:19:45.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:19:45.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxf+Bm5p8DCxAA2uCUYpjyR21kn3cMwBTmsA== --osd-uuid da4597c2-e108-47d4-bd4b-e2dde41d3564 2024-09-10T17:19:45.677 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:45.676+0000 7feef161e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:45.684 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:45.683+0000 7feef161e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:45.685 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:45.684+0000 7feef161e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:45.686 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:45.684+0000 7feef161e640 -1 bdev(0x55c976e9dc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:19:45.686 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:45.685+0000 7feef161e640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:19:51.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:19:51.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:19:51.673 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:19:51.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:19:51.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:19:52.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:19:52.232 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:19:52.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:19:52.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:19:52.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:19:52.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:19:52.264 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:52.262+0000 7fd552619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:52.280 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:52.278+0000 7fd552619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:52.281 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:52.280+0000 7fd552619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:52.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:19:52.695 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:19:52.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:52.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:19:53.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:53.636 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:53.635+0000 7fd552619640 -1 Falling back to public interface 2024-09-10T17:19:54.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:54.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:54.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:19:54.113 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:19:54.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:54.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:19:54.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:55.207 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:19:55.206+0000 7fd552619640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:19:55.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:55.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:55.522 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:19:55.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:19:55.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:55.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:19:56.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:57.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:57.026 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:19:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:19:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:57.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:19:57.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:58.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:58.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:58.453 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:19:58.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:19:58.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:58.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:19:58.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:19:59.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:19:59.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:19:59.875 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:19:59.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:19:59.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:19:59.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:00.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:01.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:01.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:01.383 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:20:01.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:20:01.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:01.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:01.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:02.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:02.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:02.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:20:02.821 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:20:02.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:02.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:03.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:04.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:04.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:04.289 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:20:04.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:20:04.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:04.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:04.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:05.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:05.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:05.775 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:20:05.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:20:05.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:05.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:06.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:07.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:07.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:07.197 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:20:07.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:20:07.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:07.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:07.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:08.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:08.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:08.613 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:20:08.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:20:08.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:08.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:09.798 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:10.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:10.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:10.800 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:20:10.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:20:10.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:10.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:11.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:12.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:12.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:12.324 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:20:12.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:20:12.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:12.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:13.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:14.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:14.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:14.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:20:14.162 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:20:14.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:14.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:14.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:15.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:15.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:15.629 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:20:15.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:20:15.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:15.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:16.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:17.282 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:20:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:20:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:18.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:19.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:19.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:19.019 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:20:19.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:20:19.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:19.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:20:19.424 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/462247432,v1:127.0.0.1:6811/462247432] [v2:127.0.0.1:6812/462247432,v1:127.0.0.1:6813/462247432] exists,up da4597c2-e108-47d4-bd4b-e2dde41d3564 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:20:19.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:20:19.429 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:20:19.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:20:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d614940-78b6-4c2e-8b4b-6a149df89842 2024-09-10T17:20:19.432 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 5d614940-78b6-4c2e-8b4b-6a149df89842 2024-09-10T17:20:19.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5d614940-78b6-4c2e-8b4b-6a149df89842' 2024-09-10T17:20:19.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:20:19.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTf+Bmb161GhAAzLvS35sCEtu6tVnMV54vjg== 2024-09-10T17:20:19.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTf+Bmb161GhAAzLvS35sCEtu6tVnMV54vjg=="}' 2024-09-10T17:20:19.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d614940-78b6-4c2e-8b4b-6a149df89842 -i td/test-erasure-eio/2/new.json 2024-09-10T17:20:20.118 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:20:20.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:20:20.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTf+Bmb161GhAAzLvS35sCEtu6tVnMV54vjg== --osd-uuid 5d614940-78b6-4c2e-8b4b-6a149df89842 2024-09-10T17:20:20.165 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:20.163+0000 7f17eb902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:20.184 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:20.183+0000 7f17eb902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:20.185 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:20.184+0000 7f17eb902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:20.186 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:20.184+0000 7f17eb902640 -1 bdev(0x55bc5ec23c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:20:20.186 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:20.185+0000 7f17eb902640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:20:25.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:20:25.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:20:25.847 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:20:25.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:20:25.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:20:26.389 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:20:26.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:20:26.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:20:26.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:20:26.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:20:26.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:20:26.421 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:26.419+0000 7fcd81f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:26.432 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:26.430+0000 7fcd81f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:26.433 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:26.431+0000 7fcd81f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:20:26.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:26.775 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:20:26.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:20:26.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:26.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:27.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:27.773 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:27.771+0000 7fcd81f45640 -1 Falling back to public interface 2024-09-10T17:20:28.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:28.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:28.205 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:20:28.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:20:28.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:28.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:28.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:29.583 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:29.582+0000 7fcd81f45640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:20:29.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:29.649 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:20:29.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:29.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:20:29.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:29.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:30.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:31.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:31.371 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:20:31.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:31.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:20:31.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:31.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:31.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:32.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:32.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:32.776 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:20:32.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:20:32.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:32.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:33.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:34.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:34.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:34.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:20:34.190 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:20:34.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:34.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:34.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:35.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:20:35.629 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:20:35.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:35.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:36.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:37.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:37.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:37.049 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:20:37.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:20:37.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:37.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:37.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:38.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:38.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:38.514 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:20:38.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:20:38.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:38.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:38.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:39.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:39.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:39.950 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:20:39.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:20:39.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:39.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:40.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:41.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:41.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:41.480 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:20:41.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:20:41.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:41.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:42.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:43.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:43.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:43.409 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:20:43.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:20:43.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:43.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:44.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:45.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:45.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:45.050 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:20:45.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:20:45.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:45.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:45.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:46.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:46.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:46.972 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:20:46.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:20:46.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:46.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:47.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:48.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:48.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:48.783 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:20:48.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:20:48.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:48.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:49.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:50.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:50.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:50.486 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:20:50.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:20:50.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:50.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:51.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:20:52.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:20:52.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:20:52.599 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:20:52.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:20:52.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:20:52.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:20:53.002 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1226459076,v1:127.0.0.1:6819/1226459076] [v2:127.0.0.1:6820/1226459076,v1:127.0.0.1:6821/1226459076] exists,up 5d614940-78b6-4c2e-8b4b-6a149df89842 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:20:53.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:20:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:20:53.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a392bee-ad07-4e76-be38-b0031dd78792 2024-09-10T17:20:53.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4a392bee-ad07-4e76-be38-b0031dd78792' 2024-09-10T17:20:53.010 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 4a392bee-ad07-4e76-be38-b0031dd78792 2024-09-10T17:20:53.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:20:53.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1f+BmpCiJARAAYcGk4/ILohjOL41FQouWWQ== 2024-09-10T17:20:53.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1f+BmpCiJARAAYcGk4/ILohjOL41FQouWWQ=="}' 2024-09-10T17:20:53.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a392bee-ad07-4e76-be38-b0031dd78792 -i td/test-erasure-eio/3/new.json 2024-09-10T17:20:53.965 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:20:53.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:20:53.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1f+BmpCiJARAAYcGk4/ILohjOL41FQouWWQ== --osd-uuid 4a392bee-ad07-4e76-be38-b0031dd78792 2024-09-10T17:20:54.010 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:54.009+0000 7f5940751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:54.012 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:54.011+0000 7f5940751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:54.013 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:54.012+0000 7f5940751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:20:54.014 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:54.012+0000 7f5940751640 -1 bdev(0x563fa7787c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:20:54.014 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:20:54.012+0000 7f5940751640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:21:00.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:21:00.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:21:00.765 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:21:00.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:21:00.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:21:01.307 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:21:01.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:21:01.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:21:01.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:21:01.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:21:01.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:21:01.339 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:01.337+0000 7fd66581e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:01.349 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:01.347+0000 7fd66581e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:01.349 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:01.348+0000 7fd66581e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:01.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:21:01.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:21:01.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:21:01.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:21:01.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:21:01.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:01.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:21:01.722 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:21:01.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:01.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:02.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:02.720 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:02.719+0000 7fd66581e640 -1 Falling back to public interface 2024-09-10T17:21:03.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:03.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:03.163 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:21:03.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:21:03.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:03.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:03.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:04.485 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:04.484+0000 7fd66581e640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:21:04.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:04.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:04.591 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:21:04.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:21:04.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:04.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:05.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:06.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:06.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:06.200 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:21:06.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:21:06.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:06.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:06.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:07.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:07.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:07.725 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:21:07.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:21:07.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:07.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:08.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:09.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:09.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:09.375 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:21:09.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:21:09.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:09.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:09.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:10.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:10.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:10.772 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:21:10.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:21:10.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:10.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:11.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:12.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:12.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:12.279 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:21:12.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:21:12.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:12.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:12.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:13.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:13.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:13.780 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:21:13.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:21:13.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:13.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:14.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:15.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:15.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:15.300 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:21:15.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:21:15.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:15.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:15.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:16.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:16.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:16.705 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:21:16.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:21:16.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:16.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:17.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:18.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:18.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:18.353 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:21:18.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:21:18.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:18.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:19.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:20.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:20.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:20.077 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:21:20.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:21:20.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:20.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:20.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:21.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:21.622 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:21:21.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:21:21.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:21.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:22.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:23.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:23.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:23.319 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:21:23.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:21:23.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:23.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:24.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:25.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:25.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:25.072 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:21:25.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:21:25.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:25.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:25.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:26.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:26.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:26.891 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:21:26.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:21:26.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:26.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:27.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:28.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:28.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:28.604 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:21:28.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:21:28.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:28.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:21:29.179 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/608270575,v1:127.0.0.1:6827/608270575] [v2:127.0.0.1:6828/608270575,v1:127.0.0.1:6829/608270575] exists,up 4a392bee-ad07-4e76-be38-b0031dd78792 2024-09-10T17:21:29.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:21:29.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:21:29.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:21:29.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:21:29.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:21:29.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=443aba13-b058-4a10-b6c3-143dae4cacfd 2024-09-10T17:21:29.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 443aba13-b058-4a10-b6c3-143dae4cacfd' 2024-09-10T17:21:29.187 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 443aba13-b058-4a10-b6c3-143dae4cacfd 2024-09-10T17:21:29.187 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:21:29.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZgOBm++YmDBAARIBKXf7PE55p5c3rGsQ0xA== 2024-09-10T17:21:29.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZgOBm++YmDBAARIBKXf7PE55p5c3rGsQ0xA=="}' 2024-09-10T17:21:29.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 443aba13-b058-4a10-b6c3-143dae4cacfd -i td/test-erasure-eio/4/new.json 2024-09-10T17:21:29.896 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:21:29.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:21:29.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZgOBm++YmDBAARIBKXf7PE55p5c3rGsQ0xA== --osd-uuid 443aba13-b058-4a10-b6c3-143dae4cacfd 2024-09-10T17:21:29.942 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:29.940+0000 7fb5a25a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:29.943 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:29.942+0000 7fb5a25a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:29.944 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:29.943+0000 7fb5a25a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:29.945 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:29.944+0000 7fb5a25a2640 -1 bdev(0x55e5cee33c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:21:29.945 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:29.944+0000 7fb5a25a2640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:21:35.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:21:35.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:21:35.909 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:21:35.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:21:35.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:21:36.509 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:21:36.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:21:36.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:21:36.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:21:36.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:21:36.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:21:36.541 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:36.540+0000 7fb85020f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:36.548 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:36.546+0000 7fb85020f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:36.549 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:36.548+0000 7fb85020f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:21:36.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:21:36.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:21:36.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:36.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:37.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:37.917 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:37.916+0000 7fb85020f640 -1 Falling back to public interface 2024-09-10T17:21:38.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:38.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:38.356 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:21:38.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:21:38.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:38.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:38.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:39.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:39.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:39.783 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:21:39.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:21:39.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:39.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:40.009 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:21:40.008+0000 7fb85020f640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:21:40.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:41.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:41.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:41.244 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:21:41.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:21:41.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:41.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:41.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:42.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:42.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:42.752 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:21:42.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:21:42.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:42.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:43.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:44.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:44.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:44.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:21:44.394 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:21:44.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:44.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:45.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:46.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:46.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:46.117 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:21:46.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:21:46.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:46.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:46.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:47.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:47.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:47.588 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:21:47.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:21:47.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:47.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:47.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:48.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:48.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:48.988 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:21:48.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:21:48.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:48.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:49.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:50.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:50.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:50.402 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:21:50.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:21:50.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:50.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:50.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:51.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:51.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:51.960 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:21:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:21:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:52.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:53.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:53.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:53.363 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:21:53.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:21:53.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:53.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:54.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:55.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:55.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:55.282 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:21:55.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:21:55.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:55.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:55.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:56.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:56.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:56.749 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:21:56.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:21:56.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:56.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:57.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:21:58.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:21:58.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:21:58.921 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:21:58.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:21:58.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:21:58.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:21:59.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:00.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:00.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:00.556 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:22:00.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:22:00.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:00.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:22:01.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:02.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:02.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:02.316 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:22:02.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:22:02.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:02.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:22:03.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:04.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:04.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:04.255 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:22:04.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:22:04.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:04.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:22:04.881 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/379088795,v1:127.0.0.1:6835/379088795] [v2:127.0.0.1:6836/379088795,v1:127.0.0.1:6837/379088795] exists,up 443aba13-b058-4a10-b6c3-143dae4cacfd 2024-09-10T17:22:04.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:22:04.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:22:04.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:22:04.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-10T17:22:04.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:22:04.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-10T17:22:04.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:22:04.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=015a1a30-6a6b-451a-8c94-33e0824e2efb 2024-09-10T17:22:04.889 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 015a1a30-6a6b-451a-8c94-33e0824e2efb 2024-09-10T17:22:04.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 015a1a30-6a6b-451a-8c94-33e0824e2efb' 2024-09-10T17:22:04.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:22:04.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8gOBmfMfzNRAA+5OFTFOALTTzlok0sWAaqQ== 2024-09-10T17:22:04.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8gOBmfMfzNRAA+5OFTFOALTTzlok0sWAaqQ=="}' 2024-09-10T17:22:04.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 015a1a30-6a6b-451a-8c94-33e0824e2efb -i td/test-erasure-eio/5/new.json 2024-09-10T17:22:05.583 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:22:05.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-10T17:22:05.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8gOBmfMfzNRAA+5OFTFOALTTzlok0sWAaqQ== --osd-uuid 015a1a30-6a6b-451a-8c94-33e0824e2efb 2024-09-10T17:22:05.629 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:05.628+0000 7f71b721a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:05.631 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:05.630+0000 7f71b721a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:05.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:05.631+0000 7f71b721a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:05.632 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:05.631+0000 7f71b721a640 -1 bdev(0x55820ebc9c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-10T17:22:05.633 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:05.631+0000 7f71b721a640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-10T17:22:13.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-10T17:22:13.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:22:13.617 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T17:22:13.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T17:22:13.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:22:14.156 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T17:22:14.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T17:22:14.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:22:14.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:22:14.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:22:14.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:22:14.187 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:14.186+0000 7fa13340b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:14.200 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:14.199+0000 7fa13340b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:14.201 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:14.200+0000 7fa13340b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:14.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:22:14.600 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:22:14.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:14.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:15.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:15.047 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:15.046+0000 7fa13340b640 -1 Falling back to public interface 2024-09-10T17:22:16.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:16.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:16.026 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:22:16.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:22:16.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:16.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:16.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:16.896 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:16.895+0000 7fa13340b640 -1 osd.5 0 log_to_monitors true 2024-09-10T17:22:17.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:17.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:17.571 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:22:17.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:22:17.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:17.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:18.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:19.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:19.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:19.287 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:22:19.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:22:19.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:19.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:19.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:20.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:20.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:20.995 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:22:20.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:22:20.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:20.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:21.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:22.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:22.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:22.546 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:22:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:22:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:22.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:23.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:24.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:24.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:24.010 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:22:24.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:22:24.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:24.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:24.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:25.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:25.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:25.563 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:22:25.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:22:25.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:25.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:25.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:26.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:26.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:26.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:22:26.966 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:22:26.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:26.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:27.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:28.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:28.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:28.381 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:22:28.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:22:28.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:28.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:29.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:30.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:30.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:30.031 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:22:30.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:22:30.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:30.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:30.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:31.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:31.430 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:22:31.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:31.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:22:31.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:31.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:31.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:32.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:32.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:32.869 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:22:32.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:22:32.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:32.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:33.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:34.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:34.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:34.609 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:22:34.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:22:34.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:34.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:35.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:36.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:36.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:36.833 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:22:36.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:22:36.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:36.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:37.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:38.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:38.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:38.373 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:22:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:22:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:38.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:38.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:39.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:39.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:39.877 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:22:39.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:22:39.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:39.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:40.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:41.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:41.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:41.654 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:22:41.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:22:41.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:41.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:22:42.114 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/552964106,v1:127.0.0.1:6843/552964106] [v2:127.0.0.1:6844/552964106,v1:127.0.0.1:6845/552964106] exists,up 015a1a30-6a6b-451a-8c94-33e0824e2efb 2024-09-10T17:22:42.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:22:42.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:22:42.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:22:42.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:22:42.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-10T17:22:42.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:22:42.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-10T17:22:42.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:22:42.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7198c6dd-c23d-45f9-a52a-a2998419919f 2024-09-10T17:22:42.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 7198c6dd-c23d-45f9-a52a-a2998419919f' 2024-09-10T17:22:42.122 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 7198c6dd-c23d-45f9-a52a-a2998419919f 2024-09-10T17:22:42.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:22:42.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBigOBm/409CBAA1lsSDlPuvL1dzkWjQ2OYYg== 2024-09-10T17:22:42.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBigOBm/409CBAA1lsSDlPuvL1dzkWjQ2OYYg=="}' 2024-09-10T17:22:42.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7198c6dd-c23d-45f9-a52a-a2998419919f -i td/test-erasure-eio/6/new.json 2024-09-10T17:22:42.980 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:22:42.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-10T17:22:42.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBigOBm/409CBAA1lsSDlPuvL1dzkWjQ2OYYg== --osd-uuid 7198c6dd-c23d-45f9-a52a-a2998419919f 2024-09-10T17:22:43.026 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:43.024+0000 7fc29cfa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:43.031 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:43.029+0000 7fc29cfa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:43.032 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:43.030+0000 7fc29cfa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:43.032 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:43.031+0000 7fc29cfa8640 -1 bdev(0x55686c57dc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-10T17:22:43.032 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:43.031+0000 7fc29cfa8640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-10T17:22:49.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-10T17:22:49.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:22:49.874 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T17:22:49.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T17:22:49.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:22:50.437 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T17:22:50.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T17:22:50.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:22:50.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:22:50.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:22:50.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:22:50.469 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:50.468+0000 7f77478ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:50.471 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:50.470+0000 7f77478ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:50.473 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:50.471+0000 7f77478ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:50.868 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:22:50.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:22:50.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:50.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:51.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:51.582 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:51.581+0000 7f77478ed640 -1 Falling back to public interface 2024-09-10T17:22:52.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:52.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:52.288 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:22:52.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:22:52.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:52.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:52.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:53.283 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:22:53.282+0000 7f77478ed640 -1 osd.6 0 log_to_monitors true 2024-09-10T17:22:53.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:53.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:53.731 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:22:53.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:22:53.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:53.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:54.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:55.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:55.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:55.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:22:55.330 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:22:55.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:55.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:55.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:56.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:56.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:56.944 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:22:56.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:22:56.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:56.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:57.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:58.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:58.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:58.455 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:22:58.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:22:58.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:58.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:22:58.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:22:59.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:22:59.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:22:59.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:22:59.922 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:22:59.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:22:59.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:00.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:01.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:01.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:01.434 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:23:01.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:23:01.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:01.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:01.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:02.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:02.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:02.837 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:23:02.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:23:02.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:02.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:03.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:04.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:04.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:04.285 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:23:04.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:23:04.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:04.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:04.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:05.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:05.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:05.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:23:05.761 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:23:05.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:05.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:06.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:07.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:07.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:07.268 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:23:07.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:23:07.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:07.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:07.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:08.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:08.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:08.777 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:23:08.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:23:08.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:08.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:09.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:10.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:10.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:10.479 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:23:10.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:23:10.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:10.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:11.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:12.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:12.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:12.359 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:23:12.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:23:12.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:12.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:13.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:14.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:14.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:14.322 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:23:14.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:23:14.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:14.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:14.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:15.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:15.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:15.982 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:23:15.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:23:15.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:15.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:16.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:17.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:17.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:17.593 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:23:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:23:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:17.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:18.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:19.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:19.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:19.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:23:19.158 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:23:19.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:19.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:23:19.752 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3496294510,v1:127.0.0.1:6851/3496294510] [v2:127.0.0.1:6852/3496294510,v1:127.0.0.1:6853/3496294510] exists,up 7198c6dd-c23d-45f9-a52a-a2998419919f 2024-09-10T17:23:19.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:23:19.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:23:19.832 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:23:19.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:23:19.844 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:19:19.766+0000 7f2955a0e640 0 load: jerasure load: lrc 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:508: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:509: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:23:19.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:23:19.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:23:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:23:20.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:23:21.391 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:23:21.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:23:22.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:23:22.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:23:22.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:23:22.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:23:22.404 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:23:22.405 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:23:22.405 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:23:22.405 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:23:22.405 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:23:22.405 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:23:22.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:23:22.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:22.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:23:23.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803823 2024-09-10T17:23:23.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803823 2024-09-10T17:23:23.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823' 2024-09-10T17:23:23.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.019 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:23:23.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607591 2024-09-10T17:23:23.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607591 2024-09-10T17:23:23.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591' 2024-09-10T17:23:23.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.214 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:23:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411367 2024-09-10T17:23:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411367 2024-09-10T17:23:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411367' 2024-09-10T17:23:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:23:23.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084323 2024-09-10T17:23:23.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084323 2024-09-10T17:23:23.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411367 3-120259084323' 2024-09-10T17:23:23.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:23:23.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691862 2024-09-10T17:23:23.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691862 2024-09-10T17:23:23.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411367 3-120259084323 4-171798691862' 2024-09-10T17:23:23.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.701 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:23:23.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332112 2024-09-10T17:23:23.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332112 2024-09-10T17:23:23.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411367 3-120259084323 4-171798691862 5-219043332112' 2024-09-10T17:23:23.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:23.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:23:23.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005065 2024-09-10T17:23:23.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005065 2024-09-10T17:23:23.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-77309411367 3-120259084323 4-171798691862 5-219043332112 6-261993005065' 2024-09-10T17:23:23.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:23.927 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803823 2024-09-10T17:23:23.927 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:23.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:23:23.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803823 2024-09-10T17:23:23.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:23.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803823 2024-09-10T17:23:23.929 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803823 2024-09-10T17:23:23.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-09-10T17:23:23.929 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:23:24.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803823 -lt 25769803823 2024-09-10T17:23:24.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:24.321 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607591 2024-09-10T17:23:24.321 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:24.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:23:24.322 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607591 2024-09-10T17:23:24.322 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:24.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607591 2024-09-10T17:23:24.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-09-10T17:23:24.323 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 51539607591 2024-09-10T17:23:24.324 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:23:24.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607591 -lt 51539607591 2024-09-10T17:23:24.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:24.781 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411367 2024-09-10T17:23:24.782 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:24.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:23:24.783 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411367 2024-09-10T17:23:24.783 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:24.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411367 2024-09-10T17:23:24.784 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411367 2024-09-10T17:23:24.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411367' 2024-09-10T17:23:24.784 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:23:25.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411367 -lt 77309411367 2024-09-10T17:23:25.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:25.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084323 2024-09-10T17:23:25.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:25.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:23:25.164 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084323 2024-09-10T17:23:25.164 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:25.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084323 2024-09-10T17:23:25.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-09-10T17:23:25.166 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084323 2024-09-10T17:23:25.166 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:23:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084323 -lt 120259084323 2024-09-10T17:23:25.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:25.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691862 2024-09-10T17:23:25.585 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:25.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:23:25.586 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691862 2024-09-10T17:23:25.586 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:25.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691862 2024-09-10T17:23:25.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691862' 2024-09-10T17:23:25.587 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691862 2024-09-10T17:23:25.587 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:23:26.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691862 -lt 171798691862 2024-09-10T17:23:26.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:26.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332112 2024-09-10T17:23:26.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:26.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:23:26.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332112 2024-09-10T17:23:26.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:26.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332112 2024-09-10T17:23:26.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332112' 2024-09-10T17:23:26.013 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 219043332112 2024-09-10T17:23:26.014 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:23:26.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332112 -lt 219043332112 2024-09-10T17:23:26.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:26.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005065 2024-09-10T17:23:26.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:26.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:23:26.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005065 2024-09-10T17:23:26.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:26.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005065 2024-09-10T17:23:26.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005065' 2024-09-10T17:23:26.421 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 261993005065 2024-09-10T17:23:26.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:23:26.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005065 -lt 261993005065 2024-09-10T17:23:26.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:23:26.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:23:26.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:23:27.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:23:27.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:23:27.777 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:23:27.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:23:27.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:23:28.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:23:28.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:23:28.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:23:28.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:23:28.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:23:28.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:23:28.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:23:28.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:23:28.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:23:28.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:23:28.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:23:28.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-10T17:23:29.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:23:29.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:23:29.132 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:23:29.132 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:23:29.132 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:23:29.132 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:23:29.132 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:23:29.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:23:29.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:23:29.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:23:29.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:23:29.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:23:29.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:23:29.554 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:23:29.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:23:29.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:23:29.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-10T17:23:29.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-10T17:23:30.219 INFO:tasks.workunit.client.0.smithi086.stderr:osd.0 is already out. 2024-09-10T17:23:30.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-10T17:23:30.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:23:30.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:23:30.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:23:30.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:23:30.232 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:23:30.232 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:23:30.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:23:30.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:23:30.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:23:30.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:23:31.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:23:31.055 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:23:31.055 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:31.056 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:23:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-10T17:23:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-10T17:23:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-10T17:23:31.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:31.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:23:31.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607595 2024-09-10T17:23:31.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607595 2024-09-10T17:23:31.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595' 2024-09-10T17:23:31.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:31.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:23:31.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411370 2024-09-10T17:23:31.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411370 2024-09-10T17:23:31.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595 2-77309411370' 2024-09-10T17:23:31.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:31.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:23:31.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084326 2024-09-10T17:23:31.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084326 2024-09-10T17:23:31.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595 2-77309411370 3-120259084326' 2024-09-10T17:23:31.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:31.897 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:23:32.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691866 2024-09-10T17:23:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691866 2024-09-10T17:23:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595 2-77309411370 3-120259084326 4-171798691866' 2024-09-10T17:23:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:32.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:23:32.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332115 2024-09-10T17:23:32.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332115 2024-09-10T17:23:32.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595 2-77309411370 3-120259084326 4-171798691866 5-219043332115' 2024-09-10T17:23:32.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:23:32.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:23:32.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=261993005068 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 261993005068 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-51539607595 2-77309411370 3-120259084326 4-171798691866 5-219043332115 6-261993005068' 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:32.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:23:32.309 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-10T17:23:32.309 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:32.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-10T17:23:32.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-10T17:23:32.310 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803827 2024-09-10T17:23:32.311 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:23:32.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-10T17:23:32.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:32.735 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607595 2024-09-10T17:23:32.735 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:32.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:23:32.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607595 2024-09-10T17:23:32.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:32.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607595 2024-09-10T17:23:32.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607595' 2024-09-10T17:23:32.737 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 51539607595 2024-09-10T17:23:32.738 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:23:33.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607595 -lt 51539607595 2024-09-10T17:23:33.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:33.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411370 2024-09-10T17:23:33.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:33.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:23:33.191 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411370 2024-09-10T17:23:33.191 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:33.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411370 2024-09-10T17:23:33.192 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411370 2024-09-10T17:23:33.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411370' 2024-09-10T17:23:33.193 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:23:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411370 -lt 77309411370 2024-09-10T17:23:33.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:33.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084326 2024-09-10T17:23:33.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:33.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:23:33.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084326 2024-09-10T17:23:33.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:33.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084326 2024-09-10T17:23:33.618 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084326 2024-09-10T17:23:33.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084326' 2024-09-10T17:23:33.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:23:34.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084326 -lt 120259084326 2024-09-10T17:23:34.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:34.052 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691866 2024-09-10T17:23:34.052 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:34.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:23:34.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691866 2024-09-10T17:23:34.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:34.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691866 2024-09-10T17:23:34.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691866' 2024-09-10T17:23:34.054 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691866 2024-09-10T17:23:34.054 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:23:34.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691866 -lt 171798691866 2024-09-10T17:23:34.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:34.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-219043332115 2024-09-10T17:23:34.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:34.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:23:34.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-219043332115 2024-09-10T17:23:34.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:34.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332115 2024-09-10T17:23:34.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 219043332115' 2024-09-10T17:23:34.522 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 219043332115 2024-09-10T17:23:34.522 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:23:34.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332116 -lt 219043332115 2024-09-10T17:23:34.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:23:34.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-261993005068 2024-09-10T17:23:34.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:23:34.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:23:34.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-261993005068 2024-09-10T17:23:34.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:23:34.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=261993005068 2024-09-10T17:23:34.973 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 261993005068 2024-09-10T17:23:34.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 261993005068' 2024-09-10T17:23:34.973 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:23:35.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 261993005069 -lt 261993005068 2024-09-10T17:23:35.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:23:35.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:23:35.359 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:23:35.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:23:35.880 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:23:36.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:23:36.257 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:23:36.257 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:23:36.257 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:23:36.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-10T17:23:36.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:23:36.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:23:36.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-10T17:23:36.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:23:36.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:23:36.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:23:36.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-10T17:23:36.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:23:36.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:23:36.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-10T17:23:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:23:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:23:36.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:23:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:23:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:23:37.473 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:23:37.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:23:38.315 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:23:38.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:23:38.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:23:38.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:38.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:23:38.327 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:38.327 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:38.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:38.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:38.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:38.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:38.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:38.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:38.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:38.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:38.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:38.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:38.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:38.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:38.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:38.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:38.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:38.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:38.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:23:38.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:38.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:38.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:38.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:23:38.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:23:38.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:23:38.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:23:38.480 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:23:38.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:23:38.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:23:38.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:23:38.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:23:38.483 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:23:38.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:38.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:23:38.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:23:38.484 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:23:38.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:23:38.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:23:38.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:23:38.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:23:38.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:38.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:38.487 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:23:38.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:23:38.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:23:38.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:23:38.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:23:38.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:38.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:38.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:23:38.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:23:38.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:23:38.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:23:38.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:23:38.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:23:38.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:38.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:23:38.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:23:39.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:23:39.173 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:23:39.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:39.256 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:39.257 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:23:39.257 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:23:39.257 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:23:39.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:23:39.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:23:40.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:23:40.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:23:40.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:23:40.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:23:40.453 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:23:40.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:23:41.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:23:41.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:23:41.465 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:23:41.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:23:41.536 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:23:41.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:23:41.545 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:23:38.985+0000 7f0847f1ec80 0 load: jerasure load: lrc 2024-09-10T17:23:41.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: local objname=myobject 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: local lastobj=100 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:635: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:638: TEST_ec_recovery_unfound: setup_osds 5 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:23:41.547 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:23:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:23:41.553 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:23:41.554 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:23:41.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a438bd7-4a03-4295-86b2-6204ce4a2b63 2024-09-10T17:23:41.555 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 4a438bd7-4a03-4295-86b2-6204ce4a2b63 2024-09-10T17:23:41.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4a438bd7-4a03-4295-86b2-6204ce4a2b63' 2024-09-10T17:23:41.555 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:23:41.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdgOBm2oxFIhAASFljFUnaBMltoTEn+/zKyQ== 2024-09-10T17:23:41.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdgOBm2oxFIhAASFljFUnaBMltoTEn+/zKyQ=="}' 2024-09-10T17:23:41.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a438bd7-4a03-4295-86b2-6204ce4a2b63 -i td/test-erasure-eio/0/new.json 2024-09-10T17:23:41.913 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:23:41.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:23:41.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdgOBm2oxFIhAASFljFUnaBMltoTEn+/zKyQ== --osd-uuid 4a438bd7-4a03-4295-86b2-6204ce4a2b63 2024-09-10T17:23:41.964 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:41.963+0000 7f5d609cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:41.981 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:41.980+0000 7f5d609cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:41.982 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:41.981+0000 7f5d609cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:41.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:41.981+0000 7f5d609cc640 -1 bdev(0x563510d2dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:23:41.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:41.981+0000 7f5d609cc640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:23:47.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:23:47.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:23:47.921 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:23:47.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:23:47.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:23:48.464 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:23:48.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:23:48.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:23:48.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:23:48.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:23:48.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:23:48.501 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:48.499+0000 7f86903dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:48.518+0000 7f86903dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:48.520 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:48.519+0000 7f86903dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:23:48.890 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:23:48.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:48.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:49.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:50.109 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:50.108+0000 7f86903dd640 -1 Falling back to public interface 2024-09-10T17:23:50.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:50.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:50.280 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:23:50.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:23:50.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:50.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:50.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:51.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:51.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:51.703 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:23:51.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:23:51.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:51.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:52.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:52.396 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:23:52.394+0000 7f86903dd640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:23:53.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:53.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:53.105 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:23:53.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:23:53.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:53.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:53.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:54.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:54.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:54.542 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:23:54.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:23:54.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:54.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:54.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:55.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:55.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:55.966 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:23:55.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:23:55.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:55.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:56.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:57.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:57.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:57.370 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:23:57.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:23:57.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:57.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:57.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:23:58.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:23:58.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:23:58.776 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:23:58.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:23:58.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:23:58.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:23:59.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:00.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:00.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:00.200 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:24:00.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:24:00.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:00.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:00.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:01.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:01.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:01.601 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:24:01.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:24:01.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:01.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:02.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:03.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:03.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:03.004 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:24:03.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:24:03.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:03.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:03.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:04.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:04.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:04.573 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:24:04.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:24:04.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:04.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:05.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:06.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:06.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:06.043 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:24:06.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:24:06.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:06.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:06.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:07.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:07.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:07.660 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:24:07.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:24:07.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:07.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:08.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:09.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:09.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:09.332 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:24:09.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:24:09.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:09.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:10.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:11.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:11.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:11.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:24:11.084 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:24:11.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:11.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:11.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:12.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:12.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:12.867 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:24:12.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:24:12.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:12.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:13.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:14.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:14.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:14.441 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:24:14.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:24:14.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:14.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:15.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:16.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:16.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:16.177 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:24:16.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:24:16.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:16.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:24:16.642 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/247772995,v1:127.0.0.1:6803/247772995] [v2:127.0.0.1:6804/247772995,v1:127.0.0.1:6805/247772995] exists,up 4a438bd7-4a03-4295-86b2-6204ce4a2b63 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:24:16.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:16.647 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:24:16.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=309aecf5-1018-4f85-81ef-25500255fc39 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 309aecf5-1018-4f85-81ef-25500255fc39' 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 309aecf5-1018-4f85-81ef-25500255fc39 2024-09-10T17:24:16.649 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:24:16.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDAgOBmHUXaJxAAxnPky+sjT5kKOg8QG98+7Q== 2024-09-10T17:24:16.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDAgOBmHUXaJxAAxnPky+sjT5kKOg8QG98+7Q=="}' 2024-09-10T17:24:16.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 309aecf5-1018-4f85-81ef-25500255fc39 -i td/test-erasure-eio/1/new.json 2024-09-10T17:24:17.147 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:24:17.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:24:17.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAgOBmHUXaJxAAxnPky+sjT5kKOg8QG98+7Q== --osd-uuid 309aecf5-1018-4f85-81ef-25500255fc39 2024-09-10T17:24:17.197 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:17.196+0000 7f75d0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.207 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:17.206+0000 7f75d0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.208 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:17.207+0000 7f75d0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:17.209 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:17.208+0000 7f75d0e19640 -1 bdev(0x5585a0b7bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:24:17.209 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:17.208+0000 7f75d0e19640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:24:22.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:24:22.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:24:22.960 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:24:22.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:24:22.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:24:23.516 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:24:23.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:24:23.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:24:23.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:24:23.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:24:23.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:24:23.553 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:23.551+0000 7fb325c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:23.736 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:23.566+0000 7fb325c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:23.736 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:23.567+0000 7fb325c1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:23.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:24:23.974 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:24:23.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:23.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:24.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:24.925 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:24.924+0000 7fb325c1d640 -1 Falling back to public interface 2024-09-10T17:24:25.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:25.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:25.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:24:25.384 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:24:25.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:25.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:25.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:26.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:26.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:26.827 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:24:26.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:24:26.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:26.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:26.863 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:26.861+0000 7fb325c1d640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:24:27.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:28.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:28.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:28.373 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:24:28.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:24:28.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:28.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:28.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:29.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:29.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:29.775 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:24:29.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:24:29.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:29.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:30.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:31.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:31.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:31.192 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:24:31.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:24:31.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:31.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:31.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:32.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:32.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:32.683 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:24:32.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:24:32.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:32.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:33.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:34.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:34.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:34.091 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:24:34.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:24:34.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:34.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:34.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:35.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:35.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:35.728 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:24:35.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:24:35.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:35.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:36.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:37.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:37.148 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:24:37.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:24:37.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:37.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:37.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:38.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:38.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:38.714 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:24:38.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:24:38.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:38.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:39.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:40.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:40.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:40.115 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:24:40.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:24:40.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:40.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:40.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:41.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:41.522 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:41.523 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:24:41.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:24:41.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:41.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:42.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:43.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:43.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:43.161 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:24:43.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:24:43.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:43.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:44.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:45.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:45.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:45.118 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:24:45.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:24:45.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:45.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:45.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:46.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:46.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:46.845 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:24:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:24:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:46.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:47.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:48.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:48.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:48.447 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:24:48.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:24:48.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:48.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:49.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:24:50.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:24:50.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:50.515 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:24:50.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:24:50.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:50.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:24:50.913 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1768948753,v1:127.0.0.1:6811/1768948753] [v2:127.0.0.1:6812/1768948753,v1:127.0.0.1:6813/1768948753] exists,up 309aecf5-1018-4f85-81ef-25500255fc39 2024-09-10T17:24:50.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:24:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:24:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:24:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=56042098-b8cc-4834-a7a6-053dc75328bb 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 56042098-b8cc-4834-a7a6-053dc75328bb' 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 56042098-b8cc-4834-a7a6-053dc75328bb 2024-09-10T17:24:50.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:24:50.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDigOBmAUMHOBAAJae5JZDF3srjPiDUoNNNCQ== 2024-09-10T17:24:50.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDigOBmAUMHOBAAJae5JZDF3srjPiDUoNNNCQ=="}' 2024-09-10T17:24:50.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 56042098-b8cc-4834-a7a6-053dc75328bb -i td/test-erasure-eio/2/new.json 2024-09-10T17:24:51.585 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:24:51.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:24:51.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDigOBmAUMHOBAAJae5JZDF3srjPiDUoNNNCQ== --osd-uuid 56042098-b8cc-4834-a7a6-053dc75328bb 2024-09-10T17:24:51.634 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:51.633+0000 7f15172b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:51.676 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:51.674+0000 7f15172b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:51.676+0000 7f15172b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:51.678 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:51.676+0000 7f15172b2640 -1 bdev(0x564d28d67c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:24:51.678 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:51.677+0000 7f15172b2640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:24:57.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:24:57.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:24:57.956 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:24:57.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:24:57.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:24:58.542 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:24:58.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:24:58.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:24:58.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:24:58.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:24:58.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:24:58.578 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:58.576+0000 7f3470e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:58.580 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:58.579+0000 7f3470e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:58.581 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:24:58.580+0000 7f3470e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:24:58.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:24:58.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:24:58.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:24:58.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:24:58.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:24:58.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:24:58.970 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:24:58.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:24:58.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:24:58.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:24:59.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:00.173 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:00.172+0000 7f3470e10640 -1 Falling back to public interface 2024-09-10T17:25:00.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:00.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:00.373 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:25:00.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:25:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:00.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:00.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:01.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:01.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:01.770 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:25:01.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:25:01.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:01.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:01.999 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:01.997+0000 7f3470e10640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:25:02.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:03.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:03.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:03.218 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:25:03.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:25:03.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:03.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:03.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:04.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:04.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:04.625 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:25:04.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:25:04.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:04.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:05.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:06.248 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:06.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:06.249 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:25:06.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:25:06.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:06.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:06.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:07.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:07.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:07.785 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:25:07.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:25:07.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:07.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:08.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:09.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:09.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:09.179 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:25:09.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:25:09.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:09.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:09.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:10.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:10.611 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:25:10.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:25:10.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:10.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:11.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:12.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:12.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:12.135 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:25:12.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:25:12.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:12.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:12.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:13.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:13.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:13.621 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:25:13.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:25:13.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:13.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:14.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:15.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:15.297 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:25:15.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:15.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:25:15.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:15.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:16.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:17.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:17.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:17.098 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:25:17.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:25:17.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:17.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:17.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:18.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:18.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:18.744 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:25:18.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:25:18.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:18.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:19.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:20.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:20.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:20.654 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:25:20.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:25:20.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:20.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:21.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:22.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:22.609 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:25:22.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:22.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:25:22.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:22.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:23.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:24.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:24.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:24.044 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:25:24.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:25:24.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:24.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:24.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:25.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:25.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:25.594 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:25:25.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:25:25.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:25.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:25:26.033 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2872961019,v1:127.0.0.1:6819/2872961019] [v2:127.0.0.1:6820/2872961019,v1:127.0.0.1:6821/2872961019] exists,up 56042098-b8cc-4834-a7a6-053dc75328bb 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:25:26.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:25:26.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:25:26.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:25:26.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e1f74ca-a014-44eb-9927-4a7e22d844df 2024-09-10T17:25:26.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0e1f74ca-a014-44eb-9927-4a7e22d844df' 2024-09-10T17:25:26.042 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 0e1f74ca-a014-44eb-9927-4a7e22d844df 2024-09-10T17:25:26.042 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:25:26.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGgeBmqJOmAxAAQbpDX4WOaSmfvqsjSBYeRg== 2024-09-10T17:25:26.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGgeBmqJOmAxAAQbpDX4WOaSmfvqsjSBYeRg=="}' 2024-09-10T17:25:26.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e1f74ca-a014-44eb-9927-4a7e22d844df -i td/test-erasure-eio/3/new.json 2024-09-10T17:25:26.934 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:25:26.945 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:25:26.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGgeBmqJOmAxAAQbpDX4WOaSmfvqsjSBYeRg== --osd-uuid 0e1f74ca-a014-44eb-9927-4a7e22d844df 2024-09-10T17:25:26.983 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:26.982+0000 7f80b1e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:26.985 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:26.984+0000 7f80b1e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:26.987 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:26.985+0000 7f80b1e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:26.987 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:26.985+0000 7f80b1e24640 -1 bdev(0x55a5550e7c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:25:26.987 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:26.985+0000 7f80b1e24640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:25:33.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:25:33.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:25:33.979 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:25:33.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:25:33.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:25:34.563 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:25:34.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:25:34.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:25:34.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:25:34.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:25:34.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:25:34.601 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:34.600+0000 7ff8a502e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:34.603 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:34.602+0000 7ff8a502e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:34.605 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:34.603+0000 7ff8a502e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:25:35.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:25:35.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:25:35.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:25:35.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:25:35.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:25:35.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:35.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:25:35.118 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:25:35.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:35.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:35.464 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:35.463+0000 7ff8a502e640 -1 Falling back to public interface 2024-09-10T17:25:35.550 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:36.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:36.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:36.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:25:36.552 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:25:36.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:36.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:36.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:37.271 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:25:37.269+0000 7ff8a502e640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:25:37.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:37.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:37.961 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:25:37.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:25:37.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:37.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:38.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:39.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:39.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:39.641 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:25:39.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:25:39.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:39.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:40.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:41.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:41.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:41.171 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:25:41.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:25:41.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:41.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:41.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:42.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:42.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:42.572 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:25:42.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:25:42.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:42.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:43.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:44.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:44.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:44.200 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:25:44.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:25:44.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:44.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:44.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:45.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:45.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:45.690 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:25:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:25:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:45.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:46.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:47.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:47.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:47.255 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:25:47.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:25:47.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:47.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:47.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:48.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:48.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:48.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:25:48.660 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:25:48.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:48.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:49.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:50.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:50.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:50.118 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:25:50.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:25:50.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:50.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:50.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:51.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:51.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:51.660 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:25:51.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:25:51.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:51.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:52.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:53.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:53.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:53.384 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:25:53.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:25:53.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:53.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:53.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:54.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:54.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:54.993 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:25:54.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:25:54.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:54.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:55.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:56.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:56.727 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:25:56.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:56.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:25:56.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:56.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:57.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:25:58.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:25:58.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:25:58.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:25:58.668 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:25:58.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:25:58.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:25:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:00.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:00.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:00.307 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:26:00.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:26:00.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:00.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:26:00.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:01.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:01.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:01.972 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:26:01.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:26:01.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:01.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:26:02.488 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/762026781,v1:127.0.0.1:6827/762026781] [v2:127.0.0.1:6828/762026781,v1:127.0.0.1:6829/762026781] exists,up 0e1f74ca-a014-44eb-9927-4a7e22d844df 2024-09-10T17:26:02.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:26:02.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:26:02.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:26:02.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:26:02.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:26:02.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:26:02.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa 2024-09-10T17:26:02.495 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa 2024-09-10T17:26:02.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa' 2024-09-10T17:26:02.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:26:02.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqgeBmHTO6HhAAdsu71dcGtBcUAfZcO1FJ7g== 2024-09-10T17:26:02.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqgeBmHTO6HhAAdsu71dcGtBcUAfZcO1FJ7g=="}' 2024-09-10T17:26:02.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa -i td/test-erasure-eio/4/new.json 2024-09-10T17:26:03.611 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:26:03.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:26:03.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqgeBmHTO6HhAAdsu71dcGtBcUAfZcO1FJ7g== --osd-uuid 2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa 2024-09-10T17:26:03.661 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:03.660+0000 7fc880a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:03.687 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:03.685+0000 7fc880a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:03.688 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:03.687+0000 7fc880a1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:03.688 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:03.687+0000 7fc880a1a640 -1 bdev(0x55d2f65bdc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:26:03.688 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:03.687+0000 7fc880a1a640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:26:10.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:26:10.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:26:10.078 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:26:10.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:26:10.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:26:10.623 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:26:10.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:26:10.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:26:10.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:26:10.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:26:10.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:26:10.660 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:10.658+0000 7f2f3760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:10.662 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:10.660+0000 7f2f3760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:10.663 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:10.661+0000 7f2f3760d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:26:11.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:26:11.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:26:11.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:26:11.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:26:11.091 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:26:11.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:26:11.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:11.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:26:11.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:11.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:11.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:11.505 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:11.504+0000 7f2f3760d640 -1 Falling back to public interface 2024-09-10T17:26:12.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:12.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:12.478 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:26:12.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:26:12.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:12.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:12.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:13.265 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:26:13.263+0000 7f2f3760d640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:26:13.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:13.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:13.886 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:26:13.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:26:13.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:13.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:14.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:15.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:15.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:15.464 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:26:15.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:26:15.464 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:15.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:16.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:17.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:17.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:17.232 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:26:17.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:26:17.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:17.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:17.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:18.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:18.640 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:26:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:26:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:18.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:19.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:20.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:20.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:20.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:26:20.072 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:26:20.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:20.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:20.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:21.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:21.598 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:26:21.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:21.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:26:21.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:21.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:22.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:23.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:23.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:23.052 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:26:23.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:26:23.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:23.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:23.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:24.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:24.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:24.458 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:26:24.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:26:24.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:24.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:24.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:25.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:25.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:25.873 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:26:25.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:26:25.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:25.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:26.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:27.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:27.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:27.615 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:26:27.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:26:27.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:27.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:28.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:29.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:29.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:29.325 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:26:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:26:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:29.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:29.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:30.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:30.851 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:26:30.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:30.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:26:30.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:30.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:31.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:32.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:32.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:32.519 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:26:32.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:26:32.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:32.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:33.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:34.434 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:26:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:26:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:35.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:36.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:36.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:36.153 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:26:36.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:26:36.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:36.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:36.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:37.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:37.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:37.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:26:37.808 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:26:37.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:37.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:38.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:26:39.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:26:39.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:26:39.233 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:26:39.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:26:39.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:26:39.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:26:39.921 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1462721793,v1:127.0.0.1:6835/1462721793] [v2:127.0.0.1:6836/1462721793,v1:127.0.0.1:6837/1462721793] exists,up 2f9c92da-f4c0-4cc4-9ec3-f2ab8c74bcaa 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:26:39.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:26:39.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:26:39.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:26:40.002 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:26:40.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:26:40.013 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:23:50.112+0000 7f86903dd640 0 load: jerasure load: lrc 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:26:40.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:26:40.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:26:40.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:26:40.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:26:40.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:26:40.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:26:40.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:26:41.982 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:26:41.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:26:42.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:26:42.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:26:42.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:26:42.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:26:42.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:26:42.996 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:26:42.996 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:26:42.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:26:42.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:26:42.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:26:43.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:26:43.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:26:43.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:26:43.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:26:43.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:26:43.084 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:4' 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:26:43.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:26:43.496 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:26:43.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-10T17:26:43.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-10T17:26:43.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-10T17:26:43.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:26:43.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:26:43.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574878 2024-09-10T17:26:43.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574878 2024-09-10T17:26:43.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574878' 2024-09-10T17:26:43.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:26:43.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:26:43.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345942 2024-09-10T17:26:43.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345942 2024-09-10T17:26:43.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574878 2-85899345942' 2024-09-10T17:26:43.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:26:43.862 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:26:43.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018894 2024-09-10T17:26:43.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018894 2024-09-10T17:26:43.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574878 2-85899345942 3-128849018894' 2024-09-10T17:26:43.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:26:43.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:26:44.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691849 2024-09-10T17:26:44.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691849 2024-09-10T17:26:44.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574878 2-85899345942 3-128849018894 4-171798691849' 2024-09-10T17:26:44.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:26:44.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-10T17:26:44.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:26:44.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:26:44.082 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-10T17:26:44.082 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:26:44.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-10T17:26:44.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-10T17:26:44.083 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803809 2024-09-10T17:26:44.083 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:26:44.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-10T17:26:44.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:26:44.467 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574878 2024-09-10T17:26:44.467 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:26:44.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:26:44.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574878 2024-09-10T17:26:44.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:26:44.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574878 2024-09-10T17:26:44.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-09-10T17:26:44.469 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574878 2024-09-10T17:26:44.469 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:26:44.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574878 -lt 55834574878 2024-09-10T17:26:44.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:26:44.899 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345942 2024-09-10T17:26:44.899 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:26:44.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:26:44.900 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345942 2024-09-10T17:26:44.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:26:44.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345942 2024-09-10T17:26:44.902 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 85899345942 2024-09-10T17:26:44.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-09-10T17:26:44.902 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:26:45.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-10T17:26:45.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:26:45.325 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018894 2024-09-10T17:26:45.325 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:26:45.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:26:45.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018894 2024-09-10T17:26:45.327 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:26:45.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018894 2024-09-10T17:26:45.328 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 128849018894 2024-09-10T17:26:45.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018894' 2024-09-10T17:26:45.328 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:26:45.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018894 -lt 128849018894 2024-09-10T17:26:45.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:26:45.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691849 2024-09-10T17:26:45.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:26:45.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:26:45.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691849 2024-09-10T17:26:45.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:26:45.754 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691849 2024-09-10T17:26:45.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691849 2024-09-10T17:26:45.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691849' 2024-09-10T17:26:45.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:26:46.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691849 -lt 171798691849 2024-09-10T17:26:46.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:26:46.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:26:46.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:26:46.678 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:26:46.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:26:46.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:26:47.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:26:47.057 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:26:47.057 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:26:47.058 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:26:47.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:26:47.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:26:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:26:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:42.904712+0000 0'0 49:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-10T17:26:41.595509+0000 0'0 2024-09-10T17:26:41.595509+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:36:59.054609+0000 0 0 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:40.909036+0000 0'0 49:23 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:51:21.158261+0000 0 0 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:40.455861+0000 0'0 49:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:14:46.544897+0000 0 0 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:41.260876+0000 0'0 49:23 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:41:19.809524+0000 0 0 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:25:27.160566+0000 0'0 49:78 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:40:39.107006+0000 0 0 2024-09-10T17:26:47.964 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:26:47.965 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:26:47.965 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:26:47.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:26:47.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:26:47.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:26:47.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:26:47.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:26:47.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:48.146 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-10T17:26:48.146 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:26:48.146 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:26:48.146 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:26:48.146 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:26:48.567 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:26:48.567 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:26:48.567 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:26:48.567 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:26:48.567 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: local last_osd=2 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:26:48.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:26:48.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:26:48.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-10T17:26:49.658 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already down. 2024-09-10T17:26:49.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-10T17:26:50.405 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already out. 2024-09-10T17:26:50.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-10T17:26:50.937 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:42.904712+0000 0'0 49:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-10T17:26:41.595509+0000 0'0 2024-09-10T17:26:41.595509+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:36:59.054609+0000 0 0 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:40.909036+0000 0'0 49:23 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:51:21.158261+0000 0 0 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:40.455861+0000 0'0 49:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:14:46.544897+0000 0 0 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:41.260876+0000 0'0 49:23 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:41:19.809524+0000 0 0 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-10T17:26:49.906489+0000 0'0 52:87 [1,0] 1 [1,0] 1 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:01:33.576744+0000 0 0 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:26:50.938 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:26:50.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:656: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-10T17:26:50.949 INFO:tasks.workunit.client.0.smithi086.stderr:4+0 records in 2024-09-10T17:26:50.949 INFO:tasks.workunit.client.0.smithi086.stderr:4+0 records out 2024-09-10T17:26:50.949 INFO:tasks.workunit.client.0.smithi086.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.0971e-05 s, 45.0 MB/s 2024-09-10T17:26:50.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: seq 1 100 2024-09-10T17:26:50.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:50.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:51.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:51.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:51.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:51.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:52.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:52.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:52.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:52.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:52.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:52.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:52.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:52.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:53.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:53.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:53.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:53.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:53.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:53.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:54.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:54.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:54.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:54.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:54.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:54.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:54.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:54.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:55.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:55.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:55.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:55.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:55.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:55.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:55.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:55.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:56.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:56.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:56.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:56.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:56.810 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:57.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:57.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:57.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:57.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:57.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:57.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:57.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:57.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:58.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:58.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:58.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:58.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:58.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:58.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:59.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:59.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:59.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:59.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:59.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:59.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-10T17:26:59.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:26:59.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:00.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:00.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:00.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:00.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:00.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:00.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:00.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:01.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:01.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:01.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:01.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:01.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:01.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:02.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:02.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:02.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:02.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:02.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:02.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:02.942 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:03.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:03.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:03.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:03.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:03.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:03.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:03.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:03.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:04.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:04.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:04.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:04.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:04.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:04.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:04.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:04.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:05.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:05.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:05.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:05.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:05.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:05.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:06.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:06.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:06.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:06.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:06.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:06.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:06.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:06.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:07.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:07.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:07.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:07.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:07.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:07.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:08.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:08.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:08.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:08.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:08.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:08.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:09.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:09.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:09.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:09.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:09.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:09.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:09.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:09.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:10.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:10.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:10.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:10.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:10.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:11.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:11.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:11.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:11.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:11.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:11.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:11.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:11.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:12.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:12.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:12.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:12.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:12.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:12.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:13.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:13.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:13.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:13.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:13.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:13.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:13.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:13.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:14.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:14.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:14.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:14.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:14.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:14.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:14.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:14.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:15.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:15.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:15.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:15.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:15.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:15.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:16.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:16.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:16.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:16.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:16.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:16.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:16.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:16.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:17.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:17.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:17.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:17.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:17.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:17.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:17.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:17.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:18.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:18.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-10T17:27:18.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:27:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:27:18.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-10T17:27:18.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:27:18.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-10T17:27:18.512 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-10T17:27:18.512 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:27:18.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:27:18.961 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:27:18.961 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:27:18.961 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr:2147483647' 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:27:18.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:27:18.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:27:18.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:27:18.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:18.965 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:18.966 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:18.966 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:27:18.966 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:27:19.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:27:19.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:27:19.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:19.053 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:19.054 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:27:19.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:27:19.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-10T17:27:19.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:663: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-10T17:27:19.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:27:19.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:27:19.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:27:19.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:27:19.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:27:19.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:27:19.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj75 2024-09-10T17:27:19.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:27:19.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:27:19.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj75 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2024-09-10T17:27:19.142 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-10T17:27:19.143 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:2147483647' 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:27:19.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:27:19.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:27:19.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:27:19.584 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T17:27:19.584 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:27:19.584 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:27:19.584 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:19.585 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:19.585 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:19.585 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:27:19.585 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:27:19.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:27:19.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:27:19.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:27:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-10T17:27:19.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:665: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-10T17:27:19.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:27:19.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:27:19.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:27:19.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:27:19.760 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:27:19.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:27:19.761 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:27:19.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:27:19.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:27:19.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:27:19.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:27:19.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:27:19.783+0000 7f554ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:27:19.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:27:19.785+0000 7f554ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:27:19.787 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:27:19.786+0000 7f554ce14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:27:20.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:27:20.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:27:20.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:27:20.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:27:20.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:27:20.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:27:21.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:27:21.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:27:21.607 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:27:21.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:27:21.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:27:21.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:27:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:27:21.650+0000 7f554ce14640 -1 Falling back to public interface 2024-09-10T17:27:22.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:27:23.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:27:23.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:27:23.027 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:27:23.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:27:23.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:27:23.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:27:23.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:27:23.781 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:27:23.779+0000 7f554ce14640 -1 osd.2 49 log_to_monitors true 2024-09-10T17:27:24.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:27:24.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:27:24.415 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:27:24.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:27:24.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:27:24.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:27:24.886 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up out weight 0 up_from 57 up_thru 22 down_at 50 last_clean_interval [20,49) [v2:127.0.0.1:6818/1058876261,v1:127.0.0.1:6819/1058876261] [v2:127.0.0.1:6820/1058876261,v1:127.0.0.1:6821/1058876261] exists,up 56042098-b8cc-4834-a7a6-053dc75328bb 2024-09-10T17:27:24.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:27:24.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:27:24.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:27:24.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-10T17:27:25.823 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already in. 2024-09-10T17:27:25.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:668: TEST_ec_recovery_unfound: sleep 15 2024-09-10T17:27:40.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: seq 1 100 2024-09-10T17:27:40.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-10T17:27:40.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: get_state 2.0 2024-09-10T17:27:40.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-10T17:27:40.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-10T17:27:40.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-10T17:27:40.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-10T17:27:41.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-10T17:27:41.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-10T17:27:41.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-10T17:27:41.219 INFO:tasks.workunit.client.0.smithi086.stdout:active+recovery_unfound+degraded 2024-09-10T17:27:41.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-10T17:27:41.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: break 2024-09-10T17:27:41.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-10T17:27:41.609 INFO:tasks.workunit.client.0.smithi086.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-10T17:27:29.687646+0000 55'101 61:356 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-10T17:26:41.595509+0000 0'0 2024-09-10T17:26:41.595509+0000 0 0 no scrub is scheduled 0 0 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:27:26.922640+0000 0'0 61:64 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T02:20:30.939561+0000 0 0 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:40.455861+0000 0'0 61:138 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:14:46.544897+0000 0 0 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:26:41.260876+0000 0'0 61:46 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:41:19.809524+0000 0 0 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T17:27:26.922712+0000 0'0 61:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T17:23:40.351836+0000 0'0 2024-09-10T17:23:40.351836+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:28:13.882390+0000 0 0 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout: 2024-09-10T17:27:41.610 INFO:tasks.workunit.client.0.smithi086.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T17:27:41.611 INFO:tasks.workunit.client.0.smithi086.stderr:dumped pgs 2024-09-10T17:27:41.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:681: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:27:41.728 INFO:tasks.workunit.client.0.smithi086.stdout:{ 2024-09-10T17:27:41.728 INFO:tasks.workunit.client.0.smithi086.stdout: "num_missing": 1, 2024-09-10T17:27:41.728 INFO:tasks.workunit.client.0.smithi086.stdout: "num_unfound": 1, 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": [ 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: "oid": { 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: "oid": "obj75", 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: "key": "", 2024-09-10T17:27:41.729 INFO:tasks.workunit.client.0.smithi086.stdout: "snapid": -2, 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: "hash": 3565005477, 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: "max": 0, 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: "pool": 2, 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: "namespace": "" 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.730 INFO:tasks.workunit.client.0.smithi086.stdout: "need": "55'76", 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "have": "0'0", 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "flags": "none", 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "locations": [ 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.731 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: ] 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "NotRecovering", 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: "available_might_have_unfound": true, 2024-09-10T17:27:41.732 INFO:tasks.workunit.client.0.smithi086.stdout: "might_have_unfound": [], 2024-09-10T17:27:41.733 INFO:tasks.workunit.client.0.smithi086.stdout: "more": false 2024-09-10T17:27:41.733 INFO:tasks.workunit.client.0.smithi086.stdout:} 2024-09-10T17:27:41.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-10T17:27:41.845 INFO:tasks.workunit.client.0.smithi086.stdout:{ 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "snap_trimq": "[]", 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "snap_trimq_len": 0, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+recovery_unfound+degraded", 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch": 61, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.846 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_recovery_backfill": [ 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "2(4)", 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:27:41.847 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "info": { 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s0", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "0", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "55'101", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "55'75", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "0'0", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 101, 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:27:41.848 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 47, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 47, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 60, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 47, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 60, 2024-09-10T17:27:41.849 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 60, 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 47, 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "55'101", 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 357, 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 61, 2024-09-10T17:27:41.850 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+recovery_unfound+degraded", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:27:29.795349+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:27:29.687646+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:27:29.795349+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:27:29.795349+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:27:26.922565+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:27:26.922565+0000", 2024-09-10T17:27:41.851 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:27:29.795349+0000", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:27:26.570385+0000", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:27:29.795349+0000", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 60, 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "0'0", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "0'0", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 47, 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:27:41.852 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 0, 2024-09-10T17:27:41.853 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "no scrub is scheduled", 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:27:41.854 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 413696, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 101, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 505, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 1, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 0, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 3, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 1, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 101, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:27:41.855 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 101, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 404, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 99, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 405504, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:27:41.856 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:27:41.857 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.858 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 100 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.859 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),4(2)", 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 1 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:27:41.860 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "peer_info": [ 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "0(3)", 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s3", 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "3", 2024-09-10T17:27:41.861 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "55'101", 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "55'101", 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "0'0", 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 101, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 47, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 47, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 60, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 47, 2024-09-10T17:27:41.862 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 60, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 60, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 47, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.863 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "55'101", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 135, 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 55, 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+undersized+degraded", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.864 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:26:49.642311+0000", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:26:49.642099+0000", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 60, 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "0'0", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "0'0", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 47, 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.865 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 0, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:27:41.866 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-12T04:32:00.627546+0000", 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 413696, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 101, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 505, 2024-09-10T17:27:41.867 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 101, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 101, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 101, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 404, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:27:41.868 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:27:41.869 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.870 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.871 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 101 2024-09-10T17:27:41.872 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:27:41.873 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "1(1)", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s1", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "1", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "55'101", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "55'101", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "0'0", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 101, 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:27:41.874 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 47, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 47, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 60, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 47, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 60, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 60, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 47, 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.875 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "55'101", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 135, 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 55, 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+undersized+degraded", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.876 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:26:49.642311+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:26:49.642099+0000", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 60, 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "0'0", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "0'0", 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 47, 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.877 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 0, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:27:41.878 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-12T04:32:00.627546+0000", 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 413696, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 101, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:27:41.879 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 505, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 1, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 101, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 101, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 101, 2024-09-10T17:27:41.880 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 404, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:27:41.881 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:27:41.882 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.883 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 101 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.884 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:27:41.885 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "2(4)", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s4", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "4", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "55'101", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "49'1", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "0'0", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 1, 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 47, 2024-09-10T17:27:41.886 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 47, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 60, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 47, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 60, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 60, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 47, 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.887 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "49'1", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 18, 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 49, 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+clean", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:26:42.904712+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.888 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:26:42.904156+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:26:42.904156+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 60, 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "0'0", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "0'0", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 47, 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:27:41.889 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 1, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 0, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 1, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:27:41.890 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-12T01:36:59.054609+0000", 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 4096, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 1, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:27:41.891 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 5, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 1, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 1, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 1, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 4, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:27:41.892 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:27:41.893 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.894 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.895 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [], 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [], 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:27:41.896 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "peer": "4(2)", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "pgid": "2.0s2", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "shared": "2", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "last_update": "55'101", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "last_complete": "55'101", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "log_tail": "0'0", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "last_user_version": 101, 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill": "MAX", 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [], 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "history": { 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_created": 47, 2024-09-10T17:27:41.897 INFO:tasks.workunit.client.0.smithi086.stdout: "epoch_pool_created": 47, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_started": 60, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_interval_clean": 47, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_split": 0, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_marked_full": 0, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "same_up_since": 60, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "same_interval_since": 60, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "same_primary_since": 47, 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.898 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "prior_readable_until_ub": 0 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "stats": { 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "version": "55'101", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_seq": 135, 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "reported_epoch": 55, 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "state": "active+undersized+degraded", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fresh": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_change": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_active": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_peered": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean": "2024-09-10T17:26:48.142834+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_active": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_became_peered": "2024-09-10T17:26:49.995283+0000", 2024-09-10T17:27:41.899 INFO:tasks.workunit.client.0.smithi086.stdout: "last_unstale": "2024-09-10T17:27:18.505843+0000", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_undegraded": "2024-09-10T17:26:49.642311+0000", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_fullsized": "2024-09-10T17:26:49.642099+0000", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "mapping_epoch": 60, 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "log_start": "0'0", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_start": "0'0", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "created": 47, 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_clean": 48, 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "parent": "0.0", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "parent_split_bits": 0, 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub": "0'0", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub": "0'0", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_deep_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.900 INFO:tasks.workunit.client.0.smithi086.stdout: "last_clean_scrub_stamp": "2024-09-10T17:26:41.595509+0000", 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_scrubbed": 0, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "log_size": 101, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "log_dups_size": 0, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "ondisk_log_size": 101, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "dirty_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "omap_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "hitset_bytes_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "pin_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "manifest_stats_invalid": false, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrimq_len": 0, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "last_scrub_duration": 0, 2024-09-10T17:27:41.901 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-12T04:32:00.627546+0000", 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_duration": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "objects_trimmed": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "snaptrim_duration": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "stat_sum": { 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes": 413696, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects": 101, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_clones": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_object_copies": 505, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_missing": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_degraded": 101, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_misplaced": 0, 2024-09-10T17:27:41.902 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_unfound": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_dirty": 101, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_whiteouts": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_read_kb": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write": 101, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_write_kb": 404, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_scrub_errors": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_deep_scrub_errors": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_recovered": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_recovered": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_keys_recovered": 0, 2024-09-10T17:27:41.903 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_omap": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_kb": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_kb": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_promote": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_high": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_flush_mode_low": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_some": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_evict_mode_full": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_pinned": 0, 2024-09-10T17:27:41.904 INFO:tasks.workunit.client.0.smithi086.stdout: "num_legacy_snapsets": 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "num_large_omap_objects": 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_manifest": 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_bytes": 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "num_omap_keys": 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "num_objects_repaired": 0 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: "up": [ 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.905 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "acting": [ 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: 3, 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: 1, 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: 4, 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: 0, 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: 2 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "avail_no_missing": [ 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "3(0)", 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "0(3)", 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "1(1)", 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: "4(2)" 2024-09-10T17:27:41.906 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "object_location_counts": [ 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": 101 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "blocked_by": [], 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "up_primary": 3, 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "acting_primary": 3, 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "purged_snaps": [] 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.907 INFO:tasks.workunit.client.0.smithi086.stdout: "empty": 0, 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "dne": 0, 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "incomplete": 0, 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "last_epoch_started": 61, 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "hit_set_history": { 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "current_last_update": "0'0", 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "history": [] 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_state": [ 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "name": "Started/Primary/Active", 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "enter_time": "2024-09-10T17:27:26.569466+0000", 2024-09-10T17:27:41.908 INFO:tasks.workunit.client.0.smithi086.stdout: "might_have_unfound": [ 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "0(3)", 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "1(1)", 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "2(4)", 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:27:41.909 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "osd": "4(2)", 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "status": "already probed" 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_progress": { 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "backfill_targets": [], 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "waiting_on_backfill": [], 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "last_backfill_started": "MIN", 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "backfill_info": { 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "begin": "MIN", 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "end": "MIN", 2024-09-10T17:27:41.910 INFO:tasks.workunit.client.0.smithi086.stdout: "objects": [] 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "peer_backfill_info": [], 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "backfills_in_flight": [], 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "recovering": [], 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "pg_backend": { 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "recovery_ops": [], 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "read_ops": [] 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: { 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "name": "Started", 2024-09-10T17:27:41.911 INFO:tasks.workunit.client.0.smithi086.stdout: "enter_time": "2024-09-10T17:27:26.044916+0000" 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: } 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: ], 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "scrubber": { 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "active": false, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "must_scrub": false, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "must_deep_scrub": false, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "must_repair": false, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "need_auto": false, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "scrub_reg_stamp": "2024-09-12T04:32:00.627546+0000", 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "schedule": "not registered for scrubbing" 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: }, 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout: "agent_state": {} 2024-09-10T17:27:41.912 INFO:tasks.workunit.client.0.smithi086.stdout:} 2024-09-10T17:27:41.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:27:41.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: grep -q obj75 2024-09-10T17:27:41.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:27:41.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-10T17:27:42.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: check=true 2024-09-10T17:27:42.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: test true == true 2024-09-10T17:27:42.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-10T17:27:42.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-10T17:27:42.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: check=0 2024-09-10T17:27:42.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:690: TEST_ec_recovery_unfound: test 0 == 0 2024-09-10T17:27:42.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-10T17:27:47.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:694: TEST_ec_recovery_unfound: test 124 = 124 2024-09-10T17:27:47.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:696: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-10T17:27:47.476 INFO:tasks.workunit.client.0.smithi086.stderr:pg has no unfound objects 2024-09-10T17:27:47.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:698: TEST_ec_recovery_unfound: wait_for_clean 2024-09-10T17:27:47.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:27:47.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:27:47.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:27:47.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:27:47.487 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:27:47.487 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:27:47.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:27:47.487 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:27:47.488 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:27:47.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:27:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:27:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:27:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:27:47.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:27:47.581 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:27:47.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:4' 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:47.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:27:48.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803824 2024-09-10T17:27:48.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803824 2024-09-10T17:27:48.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824' 2024-09-10T17:27:48.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:48.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:27:48.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574893 2024-09-10T17:27:48.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574893 2024-09-10T17:27:48.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574893' 2024-09-10T17:27:48.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:48.223 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:27:48.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-10T17:27:48.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-10T17:27:48.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574893 2-244813135879' 2024-09-10T17:27:48.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:48.336 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:27:48.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018909 2024-09-10T17:27:48.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018909 2024-09-10T17:27:48.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574893 2-244813135879 3-128849018909' 2024-09-10T17:27:48.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:27:48.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:27:48.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691863 2024-09-10T17:27:48.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691863 2024-09-10T17:27:48.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-55834574893 2-244813135879 3-128849018909 4-171798691863' 2024-09-10T17:27:48.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:48.606 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803824 2024-09-10T17:27:48.606 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:48.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:27:48.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803824 2024-09-10T17:27:48.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:48.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803824 2024-09-10T17:27:48.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-10T17:27:48.608 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803824 2024-09-10T17:27:48.608 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:27:49.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-10T17:27:49.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:49.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574893 2024-09-10T17:27:49.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:49.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:27:49.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574893 2024-09-10T17:27:49.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:49.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574893 2024-09-10T17:27:49.035 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574893 2024-09-10T17:27:49.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574893' 2024-09-10T17:27:49.036 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:27:49.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574892 -lt 55834574893 2024-09-10T17:27:49.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:27:50.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:27:50.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:27:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574893 -lt 55834574893 2024-09-10T17:27:50.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:50.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-244813135879 2024-09-10T17:27:50.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:50.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:27:50.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-244813135879 2024-09-10T17:27:50.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:50.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-10T17:27:50.835 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 244813135879 2024-09-10T17:27:50.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 244813135879' 2024-09-10T17:27:50.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:27:51.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135880 -lt 244813135879 2024-09-10T17:27:51.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:51.254 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018909 2024-09-10T17:27:51.255 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:51.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:27:51.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018909 2024-09-10T17:27:51.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:51.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018909 2024-09-10T17:27:51.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018909' 2024-09-10T17:27:51.258 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 128849018909 2024-09-10T17:27:51.258 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:27:51.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018909 -lt 128849018909 2024-09-10T17:27:51.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:27:51.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-171798691863 2024-09-10T17:27:51.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:27:51.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:27:51.681 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-171798691863 2024-09-10T17:27:51.682 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:27:51.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691863 2024-09-10T17:27:51.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-09-10T17:27:51.683 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 171798691863 2024-09-10T17:27:51.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:27:52.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691864 -lt 171798691863 2024-09-10T17:27:52.106 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:27:52.106 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:27:52.106 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:27:52.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:27:52.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:27:52.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:27:52.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:27:52.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:27:52.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:27:52.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:27:52.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:27:53.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:27:53.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:27:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:27:53.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:27:53.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:27:53.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:27:53.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:27:53.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: seq 1 100 2024-09-10T17:27:53.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-10T17:27:53.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-10T17:27:53.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-10T17:27:53.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-10T17:27:53.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-10T17:27:53.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-10T17:27:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.799 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-10T17:27:53.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-10T17:27:53.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-10T17:27:53.863 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-10T17:27:53.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-10T17:27:53.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-10T17:27:53.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-10T17:27:53.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:53.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:53.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-10T17:27:53.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-10T17:27:54.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-10T17:27:54.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-10T17:27:54.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-10T17:27:54.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-10T17:27:54.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-10T17:27:54.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-10T17:27:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-10T17:27:54.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-10T17:27:54.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-10T17:27:54.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-10T17:27:54.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-10T17:27:54.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-10T17:27:54.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-10T17:27:54.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-10T17:27:54.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-10T17:27:54.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-10T17:27:54.592 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-10T17:27:54.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-10T17:27:54.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-10T17:27:54.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-10T17:27:54.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-10T17:27:54.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-10T17:27:54.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-10T17:27:54.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-10T17:27:54.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-10T17:27:54.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-10T17:27:54.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-10T17:27:54.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-10T17:27:54.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-10T17:27:54.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:54.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:54.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-10T17:27:54.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-10T17:27:55.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.057 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-10T17:27:55.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-10T17:27:55.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-10T17:27:55.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-10T17:27:55.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-10T17:27:55.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-10T17:27:55.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-10T17:27:55.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-10T17:27:55.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-10T17:27:55.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-10T17:27:55.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-10T17:27:55.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-10T17:27:55.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-10T17:27:55.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-10T17:27:55.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-10T17:27:55.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-10T17:27:55.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-10T17:27:55.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-10T17:27:55.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-10T17:27:55.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-10T17:27:55.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-10T17:27:55.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-10T17:27:55.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-10T17:27:55.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-10T17:27:55.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-10T17:27:55.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-10T17:27:55.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-10T17:27:55.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-10T17:27:55.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-10T17:27:55.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-10T17:27:55.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-10T17:27:55.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-10T17:27:55.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:704: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.021 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-10T17:27:56.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-10T17:27:56.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-10T17:27:56.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-10T17:27:56.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-10T17:27:56.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-10T17:27:56.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-10T17:27:56.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-10T17:27:56.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-10T17:27:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-10T17:27:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-10T17:27:56.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-10T17:27:56.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-10T17:27:56.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-10T17:27:56.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-10T17:27:56.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-10T17:27:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-10T17:27:56.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-10T17:27:56.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-10T17:27:56.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-10T17:27:56.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-10T17:27:56.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-10T17:27:56.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-10T17:27:56.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-10T17:27:56.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-10T17:27:56.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-10T17:27:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-10T17:27:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-10T17:27:56.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:711: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-10T17:27:56.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:713: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:27:56.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:27:56.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:27:57.391 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:27:57.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:27:58.179 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:27:58.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:27:58.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:27:58.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:27:58.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:27:58.191 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:27:58.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:27:58.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:27:58.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:27:58.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:27:58.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:27:58.301 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:27:58.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:27:58.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:27:58.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:27:58.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:27:58.304 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:27:58.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:27:58.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:27:58.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:27:58.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:27:58.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:27:58.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:27:58.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:27:58.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:27:58.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:58.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:58.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:27:58.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:27:58.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:27:58.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:27:58.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:27:58.346 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:27:58.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:27:58.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:27:58.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:27:58.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:27:58.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:27:58.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:27:58.349 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:27:58.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:27:58.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:27:58.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:27:58.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:27:58.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:58.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:58.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:27:58.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:27:58.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:27:58.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:27:58.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:27:58.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:58.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:58.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:27:58.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:27:58.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:27:58.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:27:58.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:27:58.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:27:58.989 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:27:58.990 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:58.990 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:58.990 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:58.991 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:27:58.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:27:58.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:27:59.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:27:59.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:27:59.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:27:59.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:27:59.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:27:59.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:27:59.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:28:00.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:28:00.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:00.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:00.002 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:00.003 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:00.003 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:28:00.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:28:00.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:28:00.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:28:00.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:28:00.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:28:00.355 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:28:00.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:28:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:28:01.438 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:28:01.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:28:01.446 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:27:58.844+0000 7f9948d22c80 0 load: jerasure load: lrc 2024-09-10T17:28:01.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-10T17:28:01.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-10T17:28:01.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: local objname=myobject 2024-09-10T17:28:01.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: setup_osds 7 2024-09-10T17:28:01.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-10T17:28:01.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:28:01.448 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-10T17:28:01.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-10T17:28:01.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:28:01.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:28:01.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:28:01.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:28:01.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:28:01.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:28:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:28:01.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=59948a3d-e2c2-42dd-b81d-78e844a323d5 2024-09-10T17:28:01.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 59948a3d-e2c2-42dd-b81d-78e844a323d5' 2024-09-10T17:28:01.456 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 59948a3d-e2c2-42dd-b81d-78e844a323d5 2024-09-10T17:28:01.456 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:28:01.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChgeBmLP4aHBAAPmobD5ZRa2lO05x3WDT65g== 2024-09-10T17:28:01.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChgeBmLP4aHBAAPmobD5ZRa2lO05x3WDT65g=="}' 2024-09-10T17:28:01.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 59948a3d-e2c2-42dd-b81d-78e844a323d5 -i td/test-erasure-eio/0/new.json 2024-09-10T17:28:01.723 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:28:01.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:28:01.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChgeBmLP4aHBAAPmobD5ZRa2lO05x3WDT65g== --osd-uuid 59948a3d-e2c2-42dd-b81d-78e844a323d5 2024-09-10T17:28:01.762 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:01.761+0000 7fcc1994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:01.783 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:01.782+0000 7fcc1994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:01.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:01.783+0000 7fcc1994b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:01.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:01.783+0000 7fcc1994b640 -1 bdev(0x558697307c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:28:01.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:01.783+0000 7fcc1994b640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:28:07.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:28:07.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:28:07.728 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:28:07.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:28:07.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:28:08.268 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:28:08.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:28:08.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:28:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:28:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:28:08.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:28:08.295 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:08.294+0000 7fb092011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:08.311 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:08.309+0000 7fb092011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:08.312 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:08.311+0000 7fb092011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:08.716 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:28:08.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:28:08.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:08.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:09.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:09.659 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:09.658+0000 7fb092011640 -1 Falling back to public interface 2024-09-10T17:28:10.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:10.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:10.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:28:10.135 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:28:10.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:10.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:10.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:11.270 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:11.269+0000 7fb092011640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:28:11.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:11.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:11.553 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:28:11.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:28:11.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:11.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:12.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:13.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:13.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:13.051 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:28:13.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:28:13.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:13.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:13.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:14.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:14.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:14.445 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:28:14.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:28:14.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:14.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:14.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:15.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:15.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:15.875 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:28:15.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:28:15.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:15.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:16.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:17.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:17.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:17.381 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:28:17.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:28:17.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:17.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:17.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:18.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:18.793 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:28:18.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:18.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:28:18.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:18.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:19.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:20.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:20.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:20.238 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:28:20.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:28:20.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:20.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:20.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:21.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:21.647 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:28:21.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:21.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:28:21.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:21.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:22.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:23.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:23.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:23.100 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:28:23.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:28:23.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:23.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:23.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:24.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:24.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:24.872 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:28:24.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:28:24.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:24.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:25.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:26.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:26.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:26.633 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:28:26.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:28:26.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:26.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:27.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:28.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:28.485 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:28:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:28:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:28.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:29.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:30.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:30.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:30.329 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:28:30.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:28:30.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:30.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:30.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:31.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:31.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:31.974 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:28:31.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:28:31.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:31.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:32.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:33.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:33.950 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:28:33.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:33.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:28:33.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:33.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:28:34.429 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1250167503,v1:127.0.0.1:6803/1250167503] [v2:127.0.0.1:6804/1250167503,v1:127.0.0.1:6805/1250167503] exists,up 59948a3d-e2c2-42dd-b81d-78e844a323d5 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:28:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:28:34.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:28:34.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3328aca1-7c50-4514-8ac4-90c243695bea 2024-09-10T17:28:34.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3328aca1-7c50-4514-8ac4-90c243695bea' 2024-09-10T17:28:34.437 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 3328aca1-7c50-4514-8ac4-90c243695bea 2024-09-10T17:28:34.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:28:34.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCgeBmecbSGhAAR0VjLrMpR0D/rfoiz7N4gw== 2024-09-10T17:28:34.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCgeBmecbSGhAAR0VjLrMpR0D/rfoiz7N4gw=="}' 2024-09-10T17:28:34.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3328aca1-7c50-4514-8ac4-90c243695bea -i td/test-erasure-eio/1/new.json 2024-09-10T17:28:35.291 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:28:35.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:28:35.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCgeBmecbSGhAAR0VjLrMpR0D/rfoiz7N4gw== --osd-uuid 3328aca1-7c50-4514-8ac4-90c243695bea 2024-09-10T17:28:35.332 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:35.330+0000 7faf42e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:35.349 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:35.348+0000 7faf42e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:35.350 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:35.349+0000 7faf42e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:35.351 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:35.349+0000 7faf42e0b640 -1 bdev(0x560d37967c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:28:35.351 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:35.349+0000 7faf42e0b640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:28:41.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:28:41.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:28:41.562 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:28:41.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:28:41.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:28:42.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:28:42.101 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:28:42.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:28:42.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:28:42.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:28:42.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:28:42.129 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:42.127+0000 7f074580d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:42.154 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:42.152+0000 7f074580d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:42.155 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:42.154+0000 7f074580d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:28:42.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:42.519 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:28:42.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:28:42.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:42.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:42.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:43.256 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:43.255+0000 7f074580d640 -1 Falling back to public interface 2024-09-10T17:28:43.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:43.937 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:28:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:28:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:44.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:44.965 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:28:44.964+0000 7f074580d640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:28:45.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:45.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:45.362 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:28:45.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:28:45.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:45.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:45.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:46.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:46.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:46.834 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:28:46.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:28:46.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:46.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:47.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:48.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:48.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:48.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:28:48.251 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:28:48.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:48.251 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:48.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:49.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:49.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:49.714 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:28:49.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:28:49.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:49.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:50.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:51.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:51.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:51.230 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:28:51.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:28:51.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:51.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:51.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:52.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:52.635 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:28:52.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:52.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:28:52.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:52.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:53.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:54.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:54.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:54.107 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:28:54.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:28:54.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:54.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:54.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:55.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:55.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:55.647 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:28:55.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:28:55.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:55.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:56.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:57.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:57.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:57.256 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:28:57.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:28:57.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:57.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:57.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:28:58.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:28:58.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:28:58.792 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:28:58.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:28:58.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:28:58.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:28:59.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:00.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:00.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:00.578 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:29:00.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:29:00.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:00.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:01.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:02.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:02.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:02.456 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:29:02.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:29:02.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:02.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:03.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:04.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:04.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:04.619 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:29:04.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:29:04.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:04.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:05.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:06.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:06.373 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:06.373 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:29:06.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:29:06.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:06.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:07.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:08.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:08.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:08.541 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:29:08.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:29:08.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:08.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:29:09.061 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/432371644,v1:127.0.0.1:6811/432371644] [v2:127.0.0.1:6812/432371644,v1:127.0.0.1:6813/432371644] exists,up 3328aca1-7c50-4514-8ac4-90c243695bea 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:29:09.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:29:09.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:29:09.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=395d34fb-9bb9-4e14-ba98-e7a53c8d5187 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 395d34fb-9bb9-4e14-ba98-e7a53c8d5187' 2024-09-10T17:29:09.069 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 395d34fb-9bb9-4e14-ba98-e7a53c8d5187 2024-09-10T17:29:09.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:29:09.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDlgeBmlkztBBAAHCygk/bo3jBa97D4u9nqiw== 2024-09-10T17:29:09.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDlgeBmlkztBBAAHCygk/bo3jBa97D4u9nqiw=="}' 2024-09-10T17:29:09.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 395d34fb-9bb9-4e14-ba98-e7a53c8d5187 -i td/test-erasure-eio/2/new.json 2024-09-10T17:29:09.764 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:29:09.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:29:09.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlgeBmlkztBBAAHCygk/bo3jBa97D4u9nqiw== --osd-uuid 395d34fb-9bb9-4e14-ba98-e7a53c8d5187 2024-09-10T17:29:09.805 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:09.803+0000 7f23da61a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:09.806 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:09.805+0000 7f23da61a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:09.807 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:09.806+0000 7f23da61a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:09.808 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:09.807+0000 7f23da61a640 -1 bdev(0x56142bb69c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:29:09.808 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:09.807+0000 7f23da61a640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:29:16.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:29:16.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:29:16.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:29:16.076 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:29:16.076 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:29:16.609 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:29:16.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:29:16.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:29:16.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:29:16.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:29:16.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:29:16.637 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:16.636+0000 7fa4bc6e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:16.659+0000 7fa4bc6e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:16.661 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:16.660+0000 7fa4bc6e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:17.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:29:17.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:29:17.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:29:17.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:29:17.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:29:17.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:17.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:29:17.084 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:29:17.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:17.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:17.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:18.008 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:18.007+0000 7fa4bc6e0640 -1 Falling back to public interface 2024-09-10T17:29:18.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:18.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:18.510 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:29:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:29:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:18.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:19.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:19.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:19.901 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:29:19.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:29:19.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:19.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:20.104 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:20.102+0000 7fa4bc6e0640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:29:20.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:21.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:21.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:21.297 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:29:21.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:29:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:21.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:21.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:22.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:22.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:22.710 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:29:22.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:29:22.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:22.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:23.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:24.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:24.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:24.169 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:29:24.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:29:24.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:24.170 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:24.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:25.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:25.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:25.717 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:29:25.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:29:25.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:25.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:26.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:27.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:27.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:27.168 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:29:27.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:29:27.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:27.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:27.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:28.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:28.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:28.577 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:29:28.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:29:28.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:28.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:29.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:30.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:30.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:30.121 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:29:30.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:29:30.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:30.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:30.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:31.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:31.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:31.618 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:29:31.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:29:31.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:31.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:32.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:33.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:33.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:33.147 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:29:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:29:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:33.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:34.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:34.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:34.901 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:29:34.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:29:34.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:34.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:35.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:36.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:36.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:36.561 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:29:36.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:29:36.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:36.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:37.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:38.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:38.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:38.716 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:29:38.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:29:38.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:38.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:39.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:40.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:40.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:40.310 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:29:40.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:29:40.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:40.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:41.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:42.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:42.297 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:29:42.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:29:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:42.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:43.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:44.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:44.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:44.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:29:44.031 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:29:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:44.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:29:44.451 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4249931676,v1:127.0.0.1:6819/4249931676] [v2:127.0.0.1:6820/4249931676,v1:127.0.0.1:6821/4249931676] exists,up 395d34fb-9bb9-4e14-ba98-e7a53c8d5187 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:29:44.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:29:44.457 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:29:44.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c 2024-09-10T17:29:44.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c' 2024-09-10T17:29:44.460 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c 2024-09-10T17:29:44.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:29:44.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIguBmrQMoHBAAtMFDZmui+oUdAwo+lP8liA== 2024-09-10T17:29:44.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIguBmrQMoHBAAtMFDZmui+oUdAwo+lP8liA=="}' 2024-09-10T17:29:44.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c -i td/test-erasure-eio/3/new.json 2024-09-10T17:29:44.983 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:29:44.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:29:44.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIguBmrQMoHBAAtMFDZmui+oUdAwo+lP8liA== --osd-uuid 91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c 2024-09-10T17:29:45.024 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:45.023+0000 7fa107789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:45.026 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:45.025+0000 7fa107789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:45.027 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:45.026+0000 7fa107789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:45.027 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:45.026+0000 7fa107789640 -1 bdev(0x55fd24971c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:29:45.027 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:45.026+0000 7fa107789640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:29:51.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:29:51.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:29:51.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:29:51.412 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:29:51.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:29:51.947 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:29:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:29:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:29:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:29:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:29:51.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:29:51.975 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:51.973+0000 7f99e3c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:52.003 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:52.002+0000 7f99e3c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:52.004 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:52.003+0000 7f99e3c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:29:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:29:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:29:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:29:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:29:52.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:29:52.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:52.402 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:29:52.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:29:52.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:52.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:29:52.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:53.597 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:53.596+0000 7f99e3c16640 -1 Falling back to public interface 2024-09-10T17:29:53.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:53.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:53.782 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:29:53.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:29:53.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:53.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:29:54.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:55.173 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:29:55.172+0000 7f99e3c16640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:29:55.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:55.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:55.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:29:55.222 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:29:55.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:55.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:29:55.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:56.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:56.913 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:29:56.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:56.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:29:56.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:56.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:29:57.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:29:58.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:29:58.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:29:58.409 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:29:58.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:29:58.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:29:58.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:29:59.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:00.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:00.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:00.031 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:30:00.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:30:00.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:00.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:00.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:01.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:01.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:01.485 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:30:01.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:30:01.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:01.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:02.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:03.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:03.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:03.016 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:30:03.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:30:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:03.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:03.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:04.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:30:04.446 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:30:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:04.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:05.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:05.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:05.842 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:30:05.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:30:05.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:05.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:06.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:07.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:07.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:07.460 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:30:07.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:30:07.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:07.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:08.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:09.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:09.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:09.638 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:30:09.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:30:09.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:09.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:10.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:11.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:11.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:11.378 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:30:11.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:30:11.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:11.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:12.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:13.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:13.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:13.146 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:30:13.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:30:13.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:13.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:13.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:14.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:14.908 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:30:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:30:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:14.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:15.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:16.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:16.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:16.644 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:30:16.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:30:16.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:16.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:17.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:18.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:18.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:18.410 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:30:18.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:30:18.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:18.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:19.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:20.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:20.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:20.002 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:30:20.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:30:20.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:20.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:30:20.576 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2513039195,v1:127.0.0.1:6827/2513039195] [v2:127.0.0.1:6828/2513039195,v1:127.0.0.1:6829/2513039195] exists,up 91e3b7ed-6f4c-4b69-b831-ce9cb9353d8c 2024-09-10T17:30:20.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:30:20.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:30:20.582 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:30:20.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:30:20.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=80130127-0d2e-4a13-a415-cd125e8dd0ed 2024-09-10T17:30:20.585 INFO:tasks.workunit.client.0.smithi086.stdout:add osd4 80130127-0d2e-4a13-a415-cd125e8dd0ed 2024-09-10T17:30:20.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 80130127-0d2e-4a13-a415-cd125e8dd0ed' 2024-09-10T17:30:20.585 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:30:20.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsguBm5hSbIxAApz0MrqlLyaP+CkoMwkfTEQ== 2024-09-10T17:30:20.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsguBm5hSbIxAApz0MrqlLyaP+CkoMwkfTEQ=="}' 2024-09-10T17:30:20.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 80130127-0d2e-4a13-a415-cd125e8dd0ed -i td/test-erasure-eio/4/new.json 2024-09-10T17:30:21.121 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:30:21.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-10T17:30:21.136 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsguBm5hSbIxAApz0MrqlLyaP+CkoMwkfTEQ== --osd-uuid 80130127-0d2e-4a13-a415-cd125e8dd0ed 2024-09-10T17:30:21.162 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:21.161+0000 7fe35efb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:21.164 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:21.163+0000 7fe35efb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:21.165 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:21.164+0000 7fe35efb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:21.165 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:21.164+0000 7fe35efb2640 -1 bdev(0x556f699abc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2024-09-10T17:30:21.165 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:21.164+0000 7fe35efb2640 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-10T17:30:27.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-10T17:30:27.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:30:27.644 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd4 key to auth repository 2024-09-10T17:30:27.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T17:30:27.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:30:28.179 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.4 2024-09-10T17:30:28.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T17:30:28.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:30:28.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:30:28.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:30:28.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:30:28.207 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:28.205+0000 7f0027217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:28.227 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:28.225+0000 7f0027217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:28.228 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:28.227+0000 7f0027217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:28.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T17:30:28.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:30:28.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T17:30:28.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:30:28.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:30:28.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:28.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:30:28.625 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:30:28.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:28.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:29.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:29.593 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:29.592+0000 7f0027217640 -1 Falling back to public interface 2024-09-10T17:30:30.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:30.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:30.060 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:30:30.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:30:30.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:30.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:30.465 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:31.439 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:31.438+0000 7f0027217640 -1 osd.4 0 log_to_monitors true 2024-09-10T17:30:31.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:31.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:31.467 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:30:31.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:30:31.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:31.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:31.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:32.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:32.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:32.955 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:30:32.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:30:32.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:32.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:33.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:34.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:34.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:34.773 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:30:34.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:30:34.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:34.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:35.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:36.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:36.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:36.292 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:30:36.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:30:36.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:36.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:36.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:37.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:37.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:37.714 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:30:37.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:30:37.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:37.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:38.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:39.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:39.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:39.114 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:30:39.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:30:39.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:39.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:39.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:40.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:40.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:40.551 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:30:40.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:30:40.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:40.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:40.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:41.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:41.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:41.959 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:30:41.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:30:41.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:41.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:42.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:43.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:43.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:43.499 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:30:43.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:30:43.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:43.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:43.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:44.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:44.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:44.921 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:30:44.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:30:44.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:44.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:45.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:46.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:46.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:46.920 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:30:46.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:30:46.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:46.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:47.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:48.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:48.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:48.861 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:30:48.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:30:48.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:48.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:49.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:50.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:50.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:50.993 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:30:50.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:30:50.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:50.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:51.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:52.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:52.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:52.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:30:52.790 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:30:52.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:52.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:53.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:54.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:54.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:54.630 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:30:54.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:30:54.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:54.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:55.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:30:56.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:30:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:30:56.176 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:30:56.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:30:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:30:56.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T17:30:56.649 INFO:tasks.workunit.client.0.smithi086.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2188333313,v1:127.0.0.1:6835/2188333313] [v2:127.0.0.1:6836/2188333313,v1:127.0.0.1:6837/2188333313] exists,up 80130127-0d2e-4a13-a415-cd125e8dd0ed 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-10T17:30:56.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:30:56.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:30:56.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dcdcd3ae-5537-4e68-977c-e7754a2c4f97 2024-09-10T17:30:56.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 dcdcd3ae-5537-4e68-977c-e7754a2c4f97' 2024-09-10T17:30:56.657 INFO:tasks.workunit.client.0.smithi086.stdout:add osd5 dcdcd3ae-5537-4e68-977c-e7754a2c4f97 2024-09-10T17:30:56.657 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:30:56.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQguBmL5HwJxAA6wDkHieQpf3O2eTYgN9v+g== 2024-09-10T17:30:56.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQguBmL5HwJxAA6wDkHieQpf3O2eTYgN9v+g=="}' 2024-09-10T17:30:56.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dcdcd3ae-5537-4e68-977c-e7754a2c4f97 -i td/test-erasure-eio/5/new.json 2024-09-10T17:30:57.661 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:30:57.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-10T17:30:57.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQguBmL5HwJxAA6wDkHieQpf3O2eTYgN9v+g== --osd-uuid dcdcd3ae-5537-4e68-977c-e7754a2c4f97 2024-09-10T17:30:57.703 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:57.702+0000 7fc4aaa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:57.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:57.703+0000 7fc4aaa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:57.706 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:57.704+0000 7fc4aaa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:30:57.706 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:57.705+0000 7fc4aaa11640 -1 bdev(0x55b430c9fc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2024-09-10T17:30:57.706 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:30:57.705+0000 7fc4aaa11640 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-10T17:31:04.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-10T17:31:04.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:31:04.830 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd5 key to auth repository 2024-09-10T17:31:04.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T17:31:04.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:31:05.363 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.5 2024-09-10T17:31:05.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T17:31:05.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:31:05.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:31:05.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:31:05.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:31:05.390 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:05.389+0000 7ff0ce82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:05.413 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:05.412+0000 7ff0ce82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:05.414 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:05.413+0000 7ff0ce82c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:31:05.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:05.828 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:31:05.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:31:05.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:05.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:06.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:06.253 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:06.252+0000 7ff0ce82c640 -1 Falling back to public interface 2024-09-10T17:31:07.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:07.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:07.213 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:31:07.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:31:07.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:07.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:07.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:08.289+0000 7ff0ce82c640 -1 osd.5 0 log_to_monitors true 2024-09-10T17:31:08.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:08.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:08.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:31:08.648 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:31:08.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:08.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:09.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:10.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:10.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:10.182 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:31:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:31:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:10.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:11.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:11.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:11.807 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:31:11.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:31:11.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:11.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:12.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:13.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:13.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:13.202 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:31:13.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:31:13.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:13.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:13.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:14.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:14.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:14.642 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:31:14.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:31:14.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:14.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:15.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:16.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:16.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:16.061 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:31:16.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:31:16.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:16.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:16.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:17.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:17.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:17.502 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:31:17.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:31:17.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:17.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:17.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:18.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:18.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:18.896 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:31:18.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:31:18.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:18.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:19.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:20.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:20.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:20.432 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:31:20.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:31:20.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:20.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:20.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:21.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:21.939 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:31:21.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:21.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:31:21.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:21.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:22.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:23.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:23.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:23.572 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:31:23.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:31:23.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:23.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:24.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:25.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:25.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:25.353 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:31:25.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:31:25.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:25.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:26.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:27.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:27.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:27.107 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:31:27.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:31:27.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:27.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:27.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:28.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:28.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:28.750 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:31:28.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:31:28.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:28.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:29.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:30.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:30.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:30.582 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:31:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:31:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:31.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:32.155 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:31:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:31:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:32.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:33.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:33.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:33.569 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:31:33.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:31:33.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:33.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T17:31:34.036 INFO:tasks.workunit.client.0.smithi086.stdout:osd.5 up in weight 1 up_from 48 up_thru 50 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2821851462,v1:127.0.0.1:6843/2821851462] [v2:127.0.0.1:6844/2821851462,v1:127.0.0.1:6845/2821851462] exists,up dcdcd3ae-5537-4e68-977c-e7754a2c4f97 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-10T17:31:34.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:31:34.041 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:31:34.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ec6bfd2a-dc47-4476-a154-499378b2fb6e 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 ec6bfd2a-dc47-4476-a154-499378b2fb6e' 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stdout:add osd6 ec6bfd2a-dc47-4476-a154-499378b2fb6e 2024-09-10T17:31:34.043 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:31:34.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB2guBmsGZpAxAASy4hz5/y75ojCkUj75iYLQ== 2024-09-10T17:31:34.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB2guBmsGZpAxAASy4hz5/y75ojCkUj75iYLQ=="}' 2024-09-10T17:31:34.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ec6bfd2a-dc47-4476-a154-499378b2fb6e -i td/test-erasure-eio/6/new.json 2024-09-10T17:31:34.669 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:31:34.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-10T17:31:34.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2guBmsGZpAxAASy4hz5/y75ojCkUj75iYLQ== --osd-uuid ec6bfd2a-dc47-4476-a154-499378b2fb6e 2024-09-10T17:31:34.710 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:34.709+0000 7f1fa569c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:34.807 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:34.805+0000 7f1fa569c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:34.807 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:34.806+0000 7f1fa569c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:34.808 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:34.807+0000 7f1fa569c640 -1 bdev(0x555603b7bc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2024-09-10T17:31:34.808 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:34.807+0000 7f1fa569c640 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-10T17:31:41.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-10T17:31:41.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:31:41.552 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd6 key to auth repository 2024-09-10T17:31:41.552 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-10T17:31:41.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:31:42.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-10T17:31:42.089 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.6 2024-09-10T17:31:42.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:31:42.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:31:42.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:31:42.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:31:42.113 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:42.112+0000 7f9ba3c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:42.138 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:42.137+0000 7f9ba3c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:42.139 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:42.138+0000 7f9ba3c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:42.503 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:31:42.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:31:42.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:42.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:42.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:43.510 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:43.509+0000 7f9ba3c10640 -1 Falling back to public interface 2024-09-10T17:31:43.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:43.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:43.921 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:31:43.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:31:43.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:43.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:44.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:45.335 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:31:45.333+0000 7f9ba3c10640 -1 osd.6 0 log_to_monitors true 2024-09-10T17:31:45.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:45.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:45.365 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:31:45.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:31:45.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:45.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:46.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:47.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:47.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:47.085 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:31:47.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:31:47.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:47.086 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:47.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:48.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:48.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:48.899 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:31:48.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:31:48.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:48.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:49.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:50.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:50.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:50.573 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:31:50.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:31:50.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:50.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:51.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:52.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:52.087 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:31:52.087 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:52.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:31:52.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:52.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:52.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:53.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:53.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:53.629 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:31:53.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:31:53.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:53.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:54.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:55.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:55.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:55.030 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:31:55.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:31:55.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:55.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:55.553 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:56.554 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:56.555 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:31:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:31:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:56.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:56.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:57.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:57.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:57.949 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:31:57.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:31:57.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:57.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:31:58.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:31:59.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:31:59.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:31:59.875 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:31:59.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:31:59.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:31:59.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:01.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:02.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:02.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:02.064 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:32:02.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:32:02.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:02.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:02.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:03.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:03.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:03.989 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:32:03.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:32:03.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:03.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:04.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:05.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:05.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:05.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:32:05.727 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:32:05.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:05.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:06.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:07.412 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:32:07.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:32:07.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:07.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:08.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:09.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:09.023 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:32:09.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:09.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:32:09.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:09.024 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:09.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:10.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:10.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:10.444 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:32:10.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:32:10.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:10.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-10T17:32:11.068 INFO:tasks.workunit.client.0.smithi086.stdout:osd.6 up in weight 1 up_from 58 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3716508007,v1:127.0.0.1:6851/3716508007] [v2:127.0.0.1:6852/3716508007,v1:127.0.0.1:6853/3716508007] exists,up ec6bfd2a-dc47-4476-a154-499378b2fb6e 2024-09-10T17:32:11.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:32:11.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:32:11.148 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:32:11.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:32:11.160 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:28:09.662+0000 7fb092011640 0 load: jerasure load: lrc 2024-09-10T17:32:11.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:415: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:416: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:32:11.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-10T17:32:11.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:32:11.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-10T17:32:11.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:32:11.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:32:12.927 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:32:12.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:32:13.939 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:32:13.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:32:13.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:32:13.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:32:13.940 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:32:13.941 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:32:13.941 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:32:13.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:32:13.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:32:13.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:32:14.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:32:14.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:32:14.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:32:14.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:32:14.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:32:14.031 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:32:14.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:32:14.516 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:32:14.516 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:32:14.516 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:14.517 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:32:14.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803825 2024-09-10T17:32:14.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803825 2024-09-10T17:32:14.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825' 2024-09-10T17:32:14.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:14.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:32:14.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607594 2024-09-10T17:32:14.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607594 2024-09-10T17:32:14.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594' 2024-09-10T17:32:14.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:14.795 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:32:14.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411366 2024-09-10T17:32:14.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411366 2024-09-10T17:32:14.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594 2-77309411366' 2024-09-10T17:32:14.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:14.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:32:15.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117021 2024-09-10T17:32:15.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117021 2024-09-10T17:32:15.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594 2-77309411366 3-115964117021' 2024-09-10T17:32:15.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:15.048 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:32:15.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757274 2024-09-10T17:32:15.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757274 2024-09-10T17:32:15.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594 2-77309411366 3-115964117021 4-163208757274' 2024-09-10T17:32:15.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:15.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:32:15.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430221 2024-09-10T17:32:15.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430221 2024-09-10T17:32:15.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594 2-77309411366 3-115964117021 4-163208757274 5-206158430221' 2024-09-10T17:32:15.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:15.270 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103173 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103173 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803825 1-51539607594 2-77309411366 3-115964117021 4-163208757274 5-206158430221 6-249108103173' 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803825 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:15.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:32:15.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803825 2024-09-10T17:32:15.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:15.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803825 2024-09-10T17:32:15.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-10T17:32:15.383 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803825 2024-09-10T17:32:15.384 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:32:15.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-10T17:32:15.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:32:16.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:32:16.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:32:17.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803826 -lt 25769803825 2024-09-10T17:32:17.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:17.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607594 2024-09-10T17:32:17.192 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:17.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:32:17.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607594 2024-09-10T17:32:17.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:17.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607594 2024-09-10T17:32:17.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607594' 2024-09-10T17:32:17.195 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 51539607594 2024-09-10T17:32:17.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:32:17.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607594 -lt 51539607594 2024-09-10T17:32:17.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:17.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411366 2024-09-10T17:32:17.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:17.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:32:17.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411366 2024-09-10T17:32:17.584 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:17.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411366 2024-09-10T17:32:17.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411366' 2024-09-10T17:32:17.586 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411366 2024-09-10T17:32:17.586 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:32:17.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411366 -lt 77309411366 2024-09-10T17:32:17.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:17.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117021 2024-09-10T17:32:17.973 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:17.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:32:17.974 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117021 2024-09-10T17:32:17.975 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:17.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117021 2024-09-10T17:32:17.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117021' 2024-09-10T17:32:17.976 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117021 2024-09-10T17:32:17.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:32:18.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117021 -lt 115964117021 2024-09-10T17:32:18.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:18.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757274 2024-09-10T17:32:18.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:18.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:32:18.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757274 2024-09-10T17:32:18.397 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:18.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757274 2024-09-10T17:32:18.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757274' 2024-09-10T17:32:18.398 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757274 2024-09-10T17:32:18.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:32:18.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757275 -lt 163208757274 2024-09-10T17:32:18.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:18.788 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-206158430221 2024-09-10T17:32:18.788 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:18.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:32:18.793 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-206158430221 2024-09-10T17:32:18.793 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:18.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430221 2024-09-10T17:32:18.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 206158430221' 2024-09-10T17:32:18.793 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 206158430221 2024-09-10T17:32:18.794 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430222 -lt 206158430221 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-249108103173 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-249108103173 2024-09-10T17:32:19.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:19.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103173 2024-09-10T17:32:19.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 249108103173' 2024-09-10T17:32:19.210 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.6 seq 249108103173 2024-09-10T17:32:19.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-10T17:32:19.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103173 -lt 249108103173 2024-09-10T17:32:19.592 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:32:19.592 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:32:19.592 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:32:20.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:32:20.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:32:20.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:32:20.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:32:20.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:32:20.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:32:20.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:32:20.119 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:32:20.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:32:20.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:32:20.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:32:20.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:32:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:32:21.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:32:21.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:32:21.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:32:21.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:32:21.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-10T17:32:21.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=myobject 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:32:21.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure myobject 2024-09-10T17:32:21.263 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:32:21.263 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:32:21.263 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:32:21.263 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:32:21.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:32:21.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:32:21.652 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:32:21.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:32:21.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:32:21.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:32:21.654 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:32:21.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:32:21.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:21.740 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:21.741 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:21.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:32:21.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:32:21.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-10T17:32:21.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-10T17:32:21.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:32:21.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2024-09-10T17:32:21.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-10T17:32:21.828 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:32:22.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:32:22.224 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:32:22.224 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_single_recovery_error: local last_osd=6 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:32:22.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:32:22.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:32:22.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-10T17:32:23.056 INFO:tasks.workunit.client.0.smithi086.stderr:osd.6 is already down. 2024-09-10T17:32:23.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-10T17:32:23.717 INFO:tasks.workunit.client.0.smithi086.stderr:osd.6 is already out. 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_single_recovery_error: wait_for_clean 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:32:23.729 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:32:23.730 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:32:23.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:32:23.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:32:23.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:32:23.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:32:24.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:32:24.229 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:4 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:5 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:6' 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.230 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:32:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803829 2024-09-10T17:32:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803829 2024-09-10T17:32:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829' 2024-09-10T17:32:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:32:24.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607597 2024-09-10T17:32:24.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607597 2024-09-10T17:32:24.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-51539607597' 2024-09-10T17:32:24.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:32:24.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411370 2024-09-10T17:32:24.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411370 2024-09-10T17:32:24.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-51539607597 2-77309411370' 2024-09-10T17:32:24.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.643 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:32:24.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117025 2024-09-10T17:32:24.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117025 2024-09-10T17:32:24.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-51539607597 2-77309411370 3-115964117025' 2024-09-10T17:32:24.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.802 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T17:32:24.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757278 2024-09-10T17:32:24.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757278 2024-09-10T17:32:24.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-51539607597 2-77309411370 3-115964117025 4-163208757278' 2024-09-10T17:32:24.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:24.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T17:32:25.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430225 2024-09-10T17:32:25.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430225 2024-09-10T17:32:25.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-51539607597 2-77309411370 3-115964117025 4-163208757278 5-206158430225' 2024-09-10T17:32:25.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:32:25.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-10T17:32:25.210 INFO:tasks.workunit.client.0.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-10T17:32:25.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T17:32:25.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T17:32:25.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T17:32:25.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:25.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803829 2024-09-10T17:32:25.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:25.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:32:25.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803829 2024-09-10T17:32:25.214 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:25.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803829 2024-09-10T17:32:25.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-10T17:32:25.215 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803829 2024-09-10T17:32:25.215 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:32:25.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803827 -lt 25769803829 2024-09-10T17:32:25.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:32:26.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:32:26.640 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:32:27.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803830 -lt 25769803829 2024-09-10T17:32:27.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:27.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607597 2024-09-10T17:32:27.067 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:27.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:32:27.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607597 2024-09-10T17:32:27.068 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:27.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607597 2024-09-10T17:32:27.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607597' 2024-09-10T17:32:27.069 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 51539607597 2024-09-10T17:32:27.070 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:32:27.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607598 -lt 51539607597 2024-09-10T17:32:27.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:27.502 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411370 2024-09-10T17:32:27.502 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:27.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:32:27.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411370 2024-09-10T17:32:27.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:27.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411370 2024-09-10T17:32:27.504 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411370 2024-09-10T17:32:27.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411370' 2024-09-10T17:32:27.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:32:27.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411370 -lt 77309411370 2024-09-10T17:32:27.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:27.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117025 2024-09-10T17:32:27.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:27.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:32:27.885 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117025 2024-09-10T17:32:27.885 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:27.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117025 2024-09-10T17:32:27.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117025' 2024-09-10T17:32:27.886 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117025 2024-09-10T17:32:27.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:32:28.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117025 -lt 115964117025 2024-09-10T17:32:28.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:28.275 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-163208757278 2024-09-10T17:32:28.275 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:28.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T17:32:28.277 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-163208757278 2024-09-10T17:32:28.277 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:28.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757278 2024-09-10T17:32:28.278 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 163208757278' 2024-09-10T17:32:28.278 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.4 seq 163208757278 2024-09-10T17:32:28.278 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T17:32:28.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757279 -lt 163208757278 2024-09-10T17:32:28.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:32:28.672 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-206158430225 2024-09-10T17:32:28.672 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:32:28.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T17:32:28.673 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-206158430225 2024-09-10T17:32:28.673 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:32:28.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430225 2024-09-10T17:32:28.674 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.5 seq 206158430225 2024-09-10T17:32:28.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 206158430225' 2024-09-10T17:32:28.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T17:32:29.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430226 -lt 206158430225 2024-09-10T17:32:29.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:32:29.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:32:29.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:32:29.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:32:29.610 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:32:30.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T17:32:30.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:32:30.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:32:30.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:32:30.537 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T17:32:30.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T17:32:30.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:32:30.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T17:32:30.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:32:30.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:32:30.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:32:30.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:32:30.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:32:31.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:32:31.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:32:31.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:32:31.010 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:32:31.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:32:31.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:32:31.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:32:31.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:32:31.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-10T17:32:31.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:32:31.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:32:31.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:32:31.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:32:31.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:32:32.081 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:32:32.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:32:32.823 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:32:32.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:32:32.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:32:32.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:32:32.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:32:32.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:32:32.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:32:32.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:32:32.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:32:32.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:32:32.946 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:32:32.946 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:32:32.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:32:32.947 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:32:32.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:32:32.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:32:32.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:32:32.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:32:32.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:32:32.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:32:32.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:32:32.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:32:32.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:32:32.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:32:32.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:32:32.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:32.985 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:32.985 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:32:32.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:32:32.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:32:32.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:32:32.988 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:32:32.988 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:32:32.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:32:32.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:32:32.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:32:32.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:32:32.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:32:32.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:32:32.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:32:32.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:32:32.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:32:32.991 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:32:32.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:32:32.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:32:32.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:32:32.993 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:32:32.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:32:32.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:32:32.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:32:32.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:32:32.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:32:32.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:32.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:32.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:32:33.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:32:33.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:32:33.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:32:33.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:32:33.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:32:33.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:32:33.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:32:33.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:32:33.437 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:32:33.437 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:32:33.437 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:33.438 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:33.438 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:32:33.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:32:33.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:32:33.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:32:33.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:32:33.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:32:33.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:32:33.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:32:33.648 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:32:33.648 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:32:33.648 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:32:33.648 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:33.649 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:33.649 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:33.649 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:32:33.649 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:32:33.649 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:32:33.731 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:32:33.732 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:33.732 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:33.733 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:33.733 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:32:33.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:32:33.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:32:33.811 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:32:34.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:34.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:32:34.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:32:34.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:32:34.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:32:34.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:32:34.978 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:32:34.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:32:35.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:32:35.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:32:36.062 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:32:36.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:32:36.071 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:32:33.469+0000 7fbc73bc1c80 0 load: jerasure load: lrc 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:32:36.072 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:32:36.073 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:32:36.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:32:36.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:32:36.080 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9eb5cb21-dadc-4318-8740-6067a9ef0da8 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9eb5cb21-dadc-4318-8740-6067a9ef0da8' 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 9eb5cb21-dadc-4318-8740-6067a9ef0da8 2024-09-10T17:32:36.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:32:36.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0guBmy5eqBRAAMSTeKjpms3BY9hPrSzk70g== 2024-09-10T17:32:36.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0guBmy5eqBRAAMSTeKjpms3BY9hPrSzk70g=="}' 2024-09-10T17:32:36.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9eb5cb21-dadc-4318-8740-6067a9ef0da8 -i td/test-erasure-eio/0/new.json 2024-09-10T17:32:36.354 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:32:36.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:32:36.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0guBmy5eqBRAAMSTeKjpms3BY9hPrSzk70g== --osd-uuid 9eb5cb21-dadc-4318-8740-6067a9ef0da8 2024-09-10T17:32:36.397 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:36.392+0000 7ff584c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:36.423 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:36.422+0000 7ff584c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:36.424 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:36.423+0000 7ff584c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:36.425 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:36.424+0000 7ff584c26640 -1 bdev(0x5594527ebc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:32:36.425 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:36.424+0000 7ff584c26640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:32:42.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:32:42.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:32:42.594 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:32:42.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:32:42.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:32:43.122 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:32:43.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:32:43.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:32:43.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:32:43.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:32:43.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:32:43.150 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:43.148+0000 7f0f41416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:43.159 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:43.158+0000 7f0f41416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:43.160 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:43.159+0000 7f0f41416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:32:43.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:32:43.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:32:43.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:32:43.573 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:32:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:32:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:43.574 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:32:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:32:43.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:43.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:43.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:44.513 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:44.512+0000 7f0f41416640 -1 Falling back to public interface 2024-09-10T17:32:45.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:45.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:45.001 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:32:45.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:32:45.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:45.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:45.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:46.137 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:32:46.135+0000 7f0f41416640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:32:46.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:46.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:46.418 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:32:46.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:32:46.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:46.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:46.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:47.873 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:47.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:47.874 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:32:47.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:32:47.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:47.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:48.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:49.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:49.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:49.289 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:32:49.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:32:49.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:49.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:49.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:50.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:50.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:50.686 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:32:50.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:32:50.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:50.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:51.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:52.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:52.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:52.084 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:32:52.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:32:52.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:52.084 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:52.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:53.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:53.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:53.493 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:32:53.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:32:53.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:53.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:54.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:55.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:55.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:55.034 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:32:55.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:32:55.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:55.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:55.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:56.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:56.434 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:32:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:32:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:56.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:56.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:57.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:57.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:57.889 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:32:57.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:32:57.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:57.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:32:58.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:32:59.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:32:59.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:32:59.722 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:32:59.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:32:59.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:32:59.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:00.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:01.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:01.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:01.532 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:33:01.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:33:01.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:01.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:02.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:03.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:03.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:03.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:33:03.376 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:33:03.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:03.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:05.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:05.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:05.514 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:33:05.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:33:05.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:05.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:06.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:07.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:07.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:07.240 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:33:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:33:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:07.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:07.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:08.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:08.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:08.766 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:33:08.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:33:08.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:08.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:33:09.270 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3150633776,v1:127.0.0.1:6803/3150633776] [v2:127.0.0.1:6804/3150633776,v1:127.0.0.1:6805/3150633776] exists,up 9eb5cb21-dadc-4318-8740-6067a9ef0da8 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:33:09.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:33:09.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:33:09.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:33:09.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ac48c398-4875-4cd0-a24f-24d90a06ea1e' 2024-09-10T17:33:09.278 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:33:09.278 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:33:09.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVguBmeixcERAAga25/5ZYNfgMkcBWb757FQ== 2024-09-10T17:33:09.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVguBmeixcERAAga25/5ZYNfgMkcBWb757FQ=="}' 2024-09-10T17:33:09.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac48c398-4875-4cd0-a24f-24d90a06ea1e -i td/test-erasure-eio/1/new.json 2024-09-10T17:33:09.912 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:33:09.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:33:09.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVguBmeixcERAAga25/5ZYNfgMkcBWb757FQ== --osd-uuid ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:33:09.953 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:09.952+0000 7f4024424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:09.987 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:09.986+0000 7f4024424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:09.988 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:09.987+0000 7f4024424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:09.989 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:09.988+0000 7f4024424640 -1 bdev(0x55b9799cbc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:33:09.989 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:09.988+0000 7f4024424640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:33:16.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:33:16.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:33:16.126 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:33:16.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:33:16.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:33:16.659 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:33:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:33:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:33:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:33:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:33:16.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:33:16.686 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:16.685+0000 7f3298a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:16.707 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:16.706+0000 7f3298a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:16.708 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:16.707+0000 7f3298a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:17.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:33:17.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:33:17.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:33:17.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:17.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:17.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:18.309 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:18.308+0000 7f3298a16640 -1 Falling back to public interface 2024-09-10T17:33:18.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:18.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:18.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:33:18.517 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:33:18.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:18.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:18.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:19.894 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:19.893+0000 7f3298a16640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:33:19.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:19.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:19.913 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:33:19.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:33:19.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:19.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:20.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:21.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:21.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:21.397 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:33:21.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:33:21.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:21.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:21.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:22.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:22.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:22.857 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:33:22.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:33:22.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:22.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:23.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:24.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:24.275 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:33:24.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:33:24.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:24.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:24.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:25.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:25.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:25.714 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:33:25.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:33:25.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:25.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:26.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:27.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:27.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:27.123 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:33:27.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:33:27.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:27.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:27.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:28.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:28.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:28.717 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:33:28.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:33:28.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:28.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:29.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:30.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:30.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:30.298 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:33:30.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:33:30.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:30.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:30.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:31.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:31.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:31.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:33:31.771 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:33:31.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:31.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:32.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:33.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:33.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:33.307 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:33:33.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:33:33.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:33.308 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:34.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:35.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:35.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:35.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:33:35.192 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:33:35.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:35.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:36.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:37.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:37.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:37.015 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:33:37.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:33:37.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:37.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:37.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:38.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:38.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:38.836 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:33:38.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:33:38.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:38.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:39.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:40.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:40.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:40.242 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:33:40.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:33:40.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:40.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:41.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:42.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:42.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:42.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:33:42.022 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:33:42.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:42.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:42.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:43.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:43.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:43.834 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:33:43.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:33:43.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:43.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:33:44.382 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2425222243,v1:127.0.0.1:6811/2425222243] [v2:127.0.0.1:6812/2425222243,v1:127.0.0.1:6813/2425222243] exists,up ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:33:44.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:33:44.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:33:44.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:33:44.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=49c77dff-1593-4cb4-a476-f19303e85b82 2024-09-10T17:33:44.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 49c77dff-1593-4cb4-a476-f19303e85b82' 2024-09-10T17:33:44.391 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 49c77dff-1593-4cb4-a476-f19303e85b82 2024-09-10T17:33:44.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:33:44.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD4guBm9w0LGBAAYN3YSEH6ZXSrzP9dsa930w== 2024-09-10T17:33:44.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD4guBm9w0LGBAAYN3YSEH6ZXSrzP9dsa930w=="}' 2024-09-10T17:33:44.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 49c77dff-1593-4cb4-a476-f19303e85b82 -i td/test-erasure-eio/2/new.json 2024-09-10T17:33:44.855 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:33:44.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:33:44.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4guBm9w0LGBAAYN3YSEH6ZXSrzP9dsa930w== --osd-uuid 49c77dff-1593-4cb4-a476-f19303e85b82 2024-09-10T17:33:44.896 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:44.895+0000 7fb58fe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:44.898 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:44.897+0000 7fb58fe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:44.899 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:44.898+0000 7fb58fe33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:44.899 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:44.898+0000 7fb58fe33640 -1 bdev(0x55d3b02b1c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:33:44.899 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:44.898+0000 7fb58fe33640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:33:51.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:33:51.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:33:51.142 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:33:51.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:33:51.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:33:51.676 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:33:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:33:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:33:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:33:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:33:51.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:33:51.704 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:51.702+0000 7fe839a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:51.705 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:51.704+0000 7fe839a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:51.707 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:51.706+0000 7fe839a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:33:52.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:33:52.106 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:33:52.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:52.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:52.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:52.541 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:52.540+0000 7fe839a17640 -1 Falling back to public interface 2024-09-10T17:33:53.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:53.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:53.492 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:33:53.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:33:53.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:53.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:53.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:54.628 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:33:54.627+0000 7fe839a17640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:33:54.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:54.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:54.889 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:33:54.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:33:54.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:54.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:55.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:56.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:56.448 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:33:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:33:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:56.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:56.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:57.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:57.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:57.912 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:33:57.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:33:57.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:57.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:58.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:33:59.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:33:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:33:59.305 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:33:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:33:59.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:33:59.306 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:33:59.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:00.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:00.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:00.813 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:34:00.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:34:00.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:00.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:01.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:02.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:02.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:02.231 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:34:02.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:34:02.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:02.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:02.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:03.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:03.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:03.624 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:34:03.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:34:03.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:03.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:04.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:05.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:05.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:05.020 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:34:05.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:34:05.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:05.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:05.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:06.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:06.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:06.474 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:34:06.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:34:06.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:06.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:07.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:08.211 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:34:08.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:34:08.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:08.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:08.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:09.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:09.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:09.849 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:34:09.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:34:09.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:09.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:10.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:11.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:11.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:11.724 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:34:11.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:34:11.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:11.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:12.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:13.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:13.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:13.217 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:34:13.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:34:13.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:13.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:14.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:15.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:15.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:15.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:34:15.257 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:34:15.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:15.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:15.727 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:16.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:16.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:16.729 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:34:16.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:34:16.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:16.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:34:17.275 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1956356384,v1:127.0.0.1:6819/1956356384] [v2:127.0.0.1:6820/1956356384,v1:127.0.0.1:6821/1956356384] exists,up 49c77dff-1593-4cb4-a476-f19303e85b82 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:34:17.280 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:34:17.281 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:34:17.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0daa19ca-0796-4d94-960e-1bab306b4758' 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:34:17.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:34:17.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZg+BmB7azERAAFrDrTrVGA3hqo2/Be/eyLQ== 2024-09-10T17:34:17.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZg+BmB7azERAAFrDrTrVGA3hqo2/Be/eyLQ=="}' 2024-09-10T17:34:17.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0daa19ca-0796-4d94-960e-1bab306b4758 -i td/test-erasure-eio/3/new.json 2024-09-10T17:34:18.338 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:34:18.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:34:18.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZg+BmB7azERAAFrDrTrVGA3hqo2/Be/eyLQ== --osd-uuid 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:34:18.379 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:18.378+0000 7fa61801a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:18.381 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:18.380+0000 7fa61801a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:18.382 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:18.381+0000 7fa61801a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:18.382 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:18.381+0000 7fa61801a640 -1 bdev(0x55bccc53c800 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:34:18.383 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:18.381+0000 7fa61801a640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:34:24.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:34:24.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:34:24.665 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:34:24.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:34:24.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:34:25.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:34:25.232 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:34:25.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:34:25.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:34:25.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:34:25.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:34:25.260 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:25.258+0000 7f26130e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:25.269 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:25.268+0000 7f26130e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:25.270 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:25.269+0000 7f26130e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:34:25.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:34:25.684 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:34:25.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:25.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:26.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:26.884 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:26.883+0000 7f26130e4640 -1 Falling back to public interface 2024-09-10T17:34:27.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:27.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:27.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:34:27.065 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:34:27.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:27.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:27.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:28.471 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:34:28.469+0000 7f26130e4640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:34:28.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:28.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:28.496 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:34:28.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:34:28.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:28.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:29.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:30.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:30.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:30.176 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:34:30.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:34:30.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:30.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:30.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:31.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:31.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:31.749 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:34:31.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:34:31.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:31.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:32.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:33.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:33.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:33.249 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:34:33.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:34:33.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:33.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:33.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:34.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:34.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:34.689 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:34:34.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:34:34.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:34.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:35.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:36.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:36.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:36.194 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:34:36.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:34:36.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:36.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:36.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:37.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:37.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:37.599 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:34:37.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:34:37.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:37.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:38.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:39.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:39.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:39.013 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:34:39.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:34:39.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:39.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:39.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:40.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:40.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:40.516 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:34:40.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:34:40.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:40.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:40.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:41.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:41.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:41.911 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:34:41.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:34:41.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:41.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:42.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:43.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:43.950 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:34:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:34:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:43.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:44.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:34:45.745 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:34:45.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:45.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:46.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:47.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:47.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:47.547 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:34:47.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:34:47.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:47.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:48.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:49.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:49.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:49.333 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:34:49.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:34:49.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:49.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:49.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:50.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:50.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:50.959 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:34:50.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:34:50.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:50.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:51.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:34:52.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:34:52.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:34:52.761 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:34:52.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:34:52.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:34:52.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:34:53.171 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3362959073,v1:127.0.0.1:6827/3362959073] [v2:127.0.0.1:6828/3362959073,v1:127.0.0.1:6829/3362959073] exists,up 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:34:53.173 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:34:53.174 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:34:53.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:34:53.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:34:53.252 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:34:53.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:34:53.263 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:32:44.516+0000 7f0f41416640 0 load: jerasure load: lrc 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:34:53.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:34:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:34:54.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:34:55.284 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:34:55.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:34:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:34:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:34:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:34:56.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:34:56.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:34:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:34:56.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:34:56.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:34:56.299 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:34:56.299 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:34:56.399 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:34:56.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:34:56.784 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:34:56.784 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:34:56.784 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:34:56.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:34:56.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:56.785 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:34:56.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803804 2024-09-10T17:34:56.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803804 2024-09-10T17:34:56.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804' 2024-09-10T17:34:56.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:56.898 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:34:57.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574867 2024-09-10T17:34:57.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574867 2024-09-10T17:34:57.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574867' 2024-09-10T17:34:57.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:57.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:34:57.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378637 2024-09-10T17:34:57.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378637 2024-09-10T17:34:57.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574867 2-81604378637' 2024-09-10T17:34:57.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:34:57.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:34:57.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117002 2024-09-10T17:34:57.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117002 2024-09-10T17:34:57.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803804 1-55834574867 2-81604378637 3-115964117002' 2024-09-10T17:34:57.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:57.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803804 2024-09-10T17:34:57.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:57.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:34:57.306 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803804 2024-09-10T17:34:57.306 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:57.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803804 2024-09-10T17:34:57.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-09-10T17:34:57.307 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803804 2024-09-10T17:34:57.307 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:57.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803802 -lt 25769803804 2024-09-10T17:34:57.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:34:58.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:34:58.690 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:34:59.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803804 2024-09-10T17:34:59.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:59.144 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574867 2024-09-10T17:34:59.144 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:59.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:34:59.145 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574867 2024-09-10T17:34:59.145 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:59.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574867 2024-09-10T17:34:59.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574867' 2024-09-10T17:34:59.146 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574867 2024-09-10T17:34:59.146 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:34:59.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574867 -lt 55834574867 2024-09-10T17:34:59.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:34:59.562 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378637 2024-09-10T17:34:59.562 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:34:59.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:34:59.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378637 2024-09-10T17:34:59.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:34:59.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378637 2024-09-10T17:34:59.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2024-09-10T17:34:59.565 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378637 2024-09-10T17:34:59.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:35:00.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378638 -lt 81604378637 2024-09-10T17:35:00.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:00.005 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117002 2024-09-10T17:35:00.005 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:00.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:35:00.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117002 2024-09-10T17:35:00.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:00.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117002 2024-09-10T17:35:00.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117002' 2024-09-10T17:35:00.008 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117002 2024-09-10T17:35:00.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:35:00.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117003 -lt 115964117002 2024-09-10T17:35:00.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:35:00.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:00.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:00.986 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:01.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:35:01.359 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:01.360 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:01.360 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:328: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:329: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-10T17:35:01.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-0-10 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-0-10 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:01.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:35:01.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:01.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:35:01.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:01.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:35:01.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-0-10 td/test-erasure-eio/ORIGINAL 2024-09-10T17:35:02.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-0-10 td/test-erasure-eio 0 10 set 2024-09-10T17:35:02.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-10 2024-09-10T17:35:02.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:35:02.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:35:02.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-10 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:02.053 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:35:02.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-10T17:35:02.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:35:03.044 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:35:03.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:35:03.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-10T17:35:03.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:10 bytes copied, 4.3376e-05 s, 231 kB/s 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:03.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:03.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:03.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:03.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-10T17:35:03.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:05.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-10T17:35:05.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:35:05.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:35:05.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:35:05.894 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:35:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:35:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:35:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:35:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:35:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.3 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:35:05.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:35:05.924 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:05.923+0000 7f44e5c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:05.927 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:05.925+0000 7f44e5c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:05.927 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:05.926+0000 7f44e5c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:06.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T17:35:06.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:06.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:06.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:07.011 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:07.010+0000 7f44e5c15640 -1 Falling back to public interface 2024-09-10T17:35:07.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:07.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:07.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:35:07.766 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:07.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:07.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:08.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:09.007 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:09.006+0000 7f44e5c15640 -1 osd.3 35 log_to_monitors true 2024-09-10T17:35:09.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:09.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:09.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:35:09.202 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:09.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:09.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [27,35) [v2:127.0.0.1:6826/2085125250,v1:127.0.0.1:6827/2085125250] [v2:127.0.0.1:6828/2085125250,v1:127.0.0.1:6829/2085125250] exists,up 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:35:09.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:35:09.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:35:09.909 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:35:09.909 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:35:09.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:35:09.910 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:35:09.910 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:35:09.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:35:09.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:35:09.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:35:09.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:35:09.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:35:09.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:10.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:35:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-10T17:35:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-10T17:35:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-10T17:35:10.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:10.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:35:10.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574872 2024-09-10T17:35:10.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574872 2024-09-10T17:35:10.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574872' 2024-09-10T17:35:10.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:10.755 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:35:10.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378642 2024-09-10T17:35:10.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378642 2024-09-10T17:35:10.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574872 2-81604378642' 2024-09-10T17:35:10.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:10.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:35:10.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691843 2024-09-10T17:35:10.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691843 2024-09-10T17:35:10.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574872 2-81604378642 3-171798691843' 2024-09-10T17:35:10.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:10.979 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-10T17:35:10.979 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:10.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:35:10.980 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-10T17:35:10.980 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-10T17:35:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-10T17:35:10.981 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803809 2024-09-10T17:35:10.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:35:11.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-10T17:35:11.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:11.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574872 2024-09-10T17:35:11.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:11.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:35:11.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574872 2024-09-10T17:35:11.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:11.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574872 2024-09-10T17:35:11.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-09-10T17:35:11.365 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 55834574872 2024-09-10T17:35:11.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:35:11.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574870 -lt 55834574872 2024-09-10T17:35:11.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:35:12.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:35:12.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:35:13.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574872 -lt 55834574872 2024-09-10T17:35:13.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:13.180 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378642 2024-09-10T17:35:13.181 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:13.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:35:13.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378642 2024-09-10T17:35:13.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:13.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378642 2024-09-10T17:35:13.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378642' 2024-09-10T17:35:13.183 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378642 2024-09-10T17:35:13.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:35:13.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378642 -lt 81604378642 2024-09-10T17:35:13.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:13.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-171798691843 2024-09-10T17:35:13.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:13.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:35:13.608 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-171798691843 2024-09-10T17:35:13.608 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691843 2024-09-10T17:35:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 171798691843' 2024-09-10T17:35:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 171798691843 2024-09-10T17:35:13.609 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:35:14.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691843 -lt 171798691843 2024-09-10T17:35:14.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:35:14.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:14.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:14.558 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:14.559 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:14.559 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:14.960 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:35:14.960 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:14.960 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:14.960 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:15.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:35:15.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:35:15.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:35:15.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:35:15.482 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:35:16.126 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:35:16.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-10 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:35:16.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-0-10 td/test-erasure-eio/COPY 2024-09-10T17:35:16.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:35:16.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:35:16.264 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-10T17:35:16.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-10T17:35:16.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-0-10 td/test-erasure-eio 1 10 set 2024-09-10T17:35:16.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-10 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:35:16.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:35:16.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:16.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:35:16.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-10 2024-09-10T17:35:16.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-10 2024-09-10T17:35:16.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:35:16.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:35:17.348 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:35:17.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:35:17.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-10T17:35:17.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-10T17:35:17.361 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:35:17.361 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:35:17.361 INFO:tasks.workunit.client.0.smithi086.stderr:10 bytes copied, 4.4999e-05 s, 222 kB/s 2024-09-10T17:35:17.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:17.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:17.667 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:35:17.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:20.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:35:20.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:20.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:35:20.394 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:35:20.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:35:20.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:35:20.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:35:20.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:35:20.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:35:20.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:35:20.422 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:20.421+0000 7feb2d41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:20.424 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:20.423+0000 7feb2d41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:20.426 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:20.425+0000 7feb2d41a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:20.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:35:20.855 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:35:20.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:20.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:21.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:21.528 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:21.527+0000 7feb2d41a640 -1 Falling back to public interface 2024-09-10T17:35:22.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:22.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:22.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:35:22.268 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:22.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:22.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:22.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:23.627 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:23.626+0000 7feb2d41a640 -1 osd.1 46 log_to_monitors true 2024-09-10T17:35:23.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:23.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:23.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:35:23.660 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:23.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:23.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:24.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:25.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 47 last_clean_interval [13,46) [v2:127.0.0.1:6810/4289402322,v1:127.0.0.1:6811/4289402322] [v2:127.0.0.1:6812/4289402322,v1:127.0.0.1:6813/4289402322] exists,up ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:35:25.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:35:25.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:35:25.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:35:25.664 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:35:25.664 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:35:25.665 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:35:25.665 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:35:25.665 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:35:25.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:35:25.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:35:25.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:35:25.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:35:25.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:35:25.756 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:26.144 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:35:26.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803814 2024-09-10T17:35:26.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803814 2024-09-10T17:35:26.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814' 2024-09-10T17:35:26.256 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:26.256 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:35:26.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332099 2024-09-10T17:35:26.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332099 2024-09-10T17:35:26.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-219043332099' 2024-09-10T17:35:26.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:26.368 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:35:26.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378647 2024-09-10T17:35:26.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378647 2024-09-10T17:35:26.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-219043332099 2-81604378647' 2024-09-10T17:35:26.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:26.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:35:26.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691848 2024-09-10T17:35:26.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691848 2024-09-10T17:35:26.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803814 1-219043332099 2-81604378647 3-171798691848' 2024-09-10T17:35:26.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:26.593 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803814 2024-09-10T17:35:26.594 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:26.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:35:26.595 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803814 2024-09-10T17:35:26.595 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:26.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803814 2024-09-10T17:35:26.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-10T17:35:26.596 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803814 2024-09-10T17:35:26.596 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:35:27.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-10T17:35:27.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:27.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-219043332099 2024-09-10T17:35:27.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:27.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:35:27.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-219043332099 2024-09-10T17:35:27.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:27.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332099 2024-09-10T17:35:27.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 219043332099' 2024-09-10T17:35:27.031 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 219043332099 2024-09-10T17:35:27.031 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:35:27.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332099 -lt 219043332099 2024-09-10T17:35:27.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:27.417 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378647 2024-09-10T17:35:27.417 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:27.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:35:27.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378647 2024-09-10T17:35:27.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:27.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378647 2024-09-10T17:35:27.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378647' 2024-09-10T17:35:27.419 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378647 2024-09-10T17:35:27.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:35:27.801 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378647 -lt 81604378647 2024-09-10T17:35:27.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:27.802 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-171798691848 2024-09-10T17:35:27.802 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:27.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:35:27.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-171798691848 2024-09-10T17:35:27.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:27.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691848 2024-09-10T17:35:27.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 171798691848' 2024-09-10T17:35:27.804 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 171798691848 2024-09-10T17:35:27.804 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:35:28.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691848 -lt 171798691848 2024-09-10T17:35:28.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:35:28.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:28.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:28.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:35:28.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:28.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:28.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:28.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:28.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:28.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:28.747 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:29.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:35:29.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:29.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:29.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:29.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:35:29.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T17:35:29.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:35:29.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T17:35:29.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:35:29.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:29.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:30.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:35:30.197 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:30.197 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:30.197 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T17:35:30.712 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T17:35:30.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T17:35:31.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T17:35:31.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T17:35:31.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T17:35:31.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:35:31.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T17:35:31.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:35:31.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:31.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:31.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:35:31.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:31.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:31.819 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:32.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:35:32.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:35:32.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:35:32.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:35:32.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:35:32.969 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-10 fail 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-10 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-10T17:35:32.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-86744-0-10 td/test-erasure-eio/COPY 2024-09-10T17:35:33.140 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj-size-86744-0-10: (5) Input/output error 2024-09-10T17:35:33.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-10T17:35:33.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:35:33.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-10T17:35:33.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:35:33.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:35:33.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-10T17:35:33.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-0-0 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-0-0 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:35:33.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:35:33.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-0-0 td/test-erasure-eio/ORIGINAL 2024-09-10T17:35:33.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-0-0 td/test-erasure-eio 0 0 set 2024-09-10T17:35:33.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-0 2024-09-10T17:35:33.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:35:33.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-0 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:33.326 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-10T17:35:33.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:35:34.426 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:35:34.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:35:34.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-10T17:35:34.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-10T17:35:34.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:34.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:34.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:34.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:34.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:34.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:34.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:34.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:35:34.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:35:34.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T17:35:34.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:35:34.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-10T17:35:34.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:35:36.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:36.899 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:35:36.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:35:36.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:35:36.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.3 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:35:36.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:35:36.929 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:36.928+0000 7f9c24a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:36.932 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:36.931+0000 7f9c24a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:36.933 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:36.932+0000 7f9c24a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:37.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T17:35:37.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:37.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:37.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:38.019 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:38.018+0000 7f9c24a1b640 -1 Falling back to public interface 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:38.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:39.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:39.815 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:39.814+0000 7f9c24a1b640 -1 osd.3 58 log_to_monitors true 2024-09-10T17:35:40.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:40.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:40.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:35:40.157 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:40.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:40.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 up in weight 1 up_from 62 up_thru 51 down_at 59 last_clean_interval [40,58) [v2:127.0.0.1:6826/3145579737,v1:127.0.0.1:6827/3145579737] [v2:127.0.0.1:6828/3145579737,v1:127.0.0.1:6829/3145579737] exists,up 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:35:40.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:35:40.753 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:35:40.753 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:35:40.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:35:40.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:35:40.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:35:40.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:35:40.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:35:40.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:35:40.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:35:40.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:35:40.842 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:35:41.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:35:41.329 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:41.329 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:41.330 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:35:41.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:35:41.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:41.330 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:35:41.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803819 2024-09-10T17:35:41.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803819 2024-09-10T17:35:41.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803819' 2024-09-10T17:35:41.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:41.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:35:41.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332104 2024-09-10T17:35:41.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332104 2024-09-10T17:35:41.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803819 1-219043332104' 2024-09-10T17:35:41.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:41.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:35:41.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378652 2024-09-10T17:35:41.856 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378652 2024-09-10T17:35:41.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803819 1-219043332104 2-81604378652' 2024-09-10T17:35:41.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:41.857 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:35:41.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972355 2024-09-10T17:35:41.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972355 2024-09-10T17:35:41.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803819 1-219043332104 2-81604378652 3-266287972355' 2024-09-10T17:35:41.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:41.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803819 2024-09-10T17:35:41.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:41.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:35:41.971 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803819 2024-09-10T17:35:41.971 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:41.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803819 2024-09-10T17:35:41.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-09-10T17:35:41.972 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803819 2024-09-10T17:35:41.972 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:35:42.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803817 -lt 25769803819 2024-09-10T17:35:42.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:35:43.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:35:43.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:35:43.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803819 -lt 25769803819 2024-09-10T17:35:43.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:43.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-219043332104 2024-09-10T17:35:43.830 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:43.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:35:43.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-219043332104 2024-09-10T17:35:43.831 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:43.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332104 2024-09-10T17:35:43.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 219043332104' 2024-09-10T17:35:43.832 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 219043332104 2024-09-10T17:35:43.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:35:44.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332104 -lt 219043332104 2024-09-10T17:35:44.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:44.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378652 2024-09-10T17:35:44.223 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:44.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:35:44.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378652 2024-09-10T17:35:44.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:44.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378652 2024-09-10T17:35:44.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378652' 2024-09-10T17:35:44.225 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378652 2024-09-10T17:35:44.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:35:44.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378652 -lt 81604378652 2024-09-10T17:35:44.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:44.641 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972355 2024-09-10T17:35:44.641 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:44.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:35:44.643 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972355 2024-09-10T17:35:44.643 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:44.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972355 2024-09-10T17:35:44.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-09-10T17:35:44.644 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 266287972355 2024-09-10T17:35:44.644 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:35:45.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972355 -lt 266287972355 2024-09-10T17:35:45.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:35:45.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:45.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:35:45.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:35:45.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:35:45.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:35:45.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:35:45.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:35:45.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:35:46.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:35:46.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:35:46.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:35:46.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:35:46.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:35:47.234 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-0 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:35:47.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-0-0 td/test-erasure-eio/COPY 2024-09-10T17:35:47.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:35:47.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:35:47.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-10T17:35:47.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-10T17:35:47.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-0-0 td/test-erasure-eio 1 0 set 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-0 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:47.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:35:47.423 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-0 2024-09-10T17:35:47.423 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-0 2024-09-10T17:35:47.423 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:35:47.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:35:47.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:35:47.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:35:48.505 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:35:48.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:35:48.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-10T17:35:48.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:35:48.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:35:48.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:35:49.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:35:49.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:49.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:35:49.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:35:49.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:35:49.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:35:49.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:35:49.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:35:51.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:35:51.932 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:35:51.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:35:51.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:35:51.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:35:51.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:35:51.934 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:35:51.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:35:51.961 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:51.960+0000 7f014f00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:51.964 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:51.962+0000 7f014f00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:51.965 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:51.964+0000 7f014f00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:35:52.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:52.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:52.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:53.551 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:53.550+0000 7f014f00e640 -1 Falling back to public interface 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:53.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:54.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:55.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:55.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:55.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:35:55.161 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:55.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:55.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:55.249 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:35:55.247+0000 7f014f00e640 -1 osd.1 67 log_to_monitors true 2024-09-10T17:35:55.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:35:56.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:35:57.192 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [51,67) [v2:127.0.0.1:6810/2216763611,v1:127.0.0.1:6811/2216763611] [v2:127.0.0.1:6812/2216763611,v1:127.0.0.1:6813/2216763611] exists,up ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:35:57.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:35:57.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:35:57.193 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:35:57.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:35:57.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:35:57.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:35:57.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:35:57.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:35:57.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:35:57.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:35:57.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:35:57.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:35:57.736 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:35:57.736 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:35:57.736 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:35:57.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:35:57.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:57.737 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:35:57.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803824 2024-09-10T17:35:57.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803824 2024-09-10T17:35:57.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824' 2024-09-10T17:35:57.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:57.849 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:35:57.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-10T17:35:57.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-10T17:35:57.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-313532612611' 2024-09-10T17:35:57.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:57.978 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:35:58.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378657 2024-09-10T17:35:58.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378657 2024-09-10T17:35:58.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-313532612611 2-81604378657' 2024-09-10T17:35:58.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:35:58.091 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:35:58.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=266287972360 2024-09-10T17:35:58.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 266287972360 2024-09-10T17:35:58.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803824 1-313532612611 2-81604378657 3-266287972360' 2024-09-10T17:35:58.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:35:58.203 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803824 2024-09-10T17:35:58.203 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:35:58.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:35:58.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803824 2024-09-10T17:35:58.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:35:58.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803824 2024-09-10T17:35:58.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-10T17:35:58.205 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803824 2024-09-10T17:35:58.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:35:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803822 -lt 25769803824 2024-09-10T17:35:58.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:35:59.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:35:59.624 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:00.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-10T17:36:00.058 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:00.058 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-313532612611 2024-09-10T17:36:00.058 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:00.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:36:00.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-313532612611 2024-09-10T17:36:00.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:00.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-10T17:36:00.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2024-09-10T17:36:00.061 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 313532612611 2024-09-10T17:36:00.061 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:00.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-10T17:36:00.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:00.476 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378657 2024-09-10T17:36:00.477 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:00.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:36:00.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378657 2024-09-10T17:36:00.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:00.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378657 2024-09-10T17:36:00.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378657' 2024-09-10T17:36:00.479 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378657 2024-09-10T17:36:00.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:36:00.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378658 -lt 81604378657 2024-09-10T17:36:00.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:00.869 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-266287972360 2024-09-10T17:36:00.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:00.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:36:00.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-266287972360 2024-09-10T17:36:00.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:00.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=266287972360 2024-09-10T17:36:00.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2024-09-10T17:36:00.872 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 266287972360 2024-09-10T17:36:00.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:36:01.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 266287972361 -lt 266287972360 2024-09-10T17:36:01.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:36:01.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:01.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:36:01.778 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:36:02.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:36:02.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:36:02.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:02.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:02.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:36:02.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:36:02.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:36:02.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:36:02.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:36:03.440 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-0 fail 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-0 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-10T17:36:03.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-10T17:36:03.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-86744-0-0 td/test-erasure-eio/COPY 2024-09-10T17:36:03.609 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj-size-86744-0-0: (5) Input/output error 2024-09-10T17:36:03.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-10T17:36:03.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:36:03.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-10T17:36:03.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:36:03.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:36:03.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-0-256 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-0-256 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:36:03.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:36:03.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-0-256 td/test-erasure-eio/ORIGINAL 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-0-256 td/test-erasure-eio 0 256 add 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-256 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:36:03.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-256 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:03.780 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:36:04.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2024-09-10T17:36:04.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:36:04.997 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:36:05.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-10T17:36:05.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:05.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:36:05.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:36:05.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:05.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:36:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:36:05.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T17:36:05.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:36:05.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-10T17:36:05.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:36:06.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:06.011 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:36:06.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.3 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:36:06.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:36:06.036 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:06.034+0000 7fb4e7b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:06.081 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:06.080+0000 7fb4e7b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:06.082 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:06.081+0000 7fb4e7b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:06.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T17:36:06.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:36:06.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:06.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:06.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:07.421 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:07.420+0000 7fb4e7b58640 -1 Falling back to public interface 2024-09-10T17:36:07.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:07.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:07.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:36:07.820 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:07.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:07.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:08.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:09.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:09.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:09.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:36:09.265 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:09.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:09.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:09.293 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:09.292+0000 7fb4e7b58640 -1 osd.3 78 log_to_monitors true 2024-09-10T17:36:09.764 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 up in weight 1 up_from 84 up_thru 74 down_at 81 last_clean_interval [62,78) [v2:127.0.0.1:6826/2727087996,v1:127.0.0.1:6827/2727087996] [v2:127.0.0.1:6828/2727087996,v1:127.0.0.1:6829/2727087996] exists,up 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:36:09.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:36:09.765 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:36:09.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:36:09.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:36:09.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:36:09.855 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:10.372 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:36:10.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803829 2024-09-10T17:36:10.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803829 2024-09-10T17:36:10.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829' 2024-09-10T17:36:10.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:10.618 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:36:10.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612616 2024-09-10T17:36:10.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612616 2024-09-10T17:36:10.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-313532612616' 2024-09-10T17:36:10.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:10.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:36:11.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378662 2024-09-10T17:36:11.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378662 2024-09-10T17:36:11.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-313532612616 2-81604378662' 2024-09-10T17:36:11.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:11.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:36:11.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252867 2024-09-10T17:36:11.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252867 2024-09-10T17:36:11.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803829 1-313532612616 2-81604378662 3-360777252867' 2024-09-10T17:36:11.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:11.130 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803829 2024-09-10T17:36:11.130 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:11.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:36:11.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803829 2024-09-10T17:36:11.131 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:11.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803829 2024-09-10T17:36:11.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-10T17:36:11.132 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803829 2024-09-10T17:36:11.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:11.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803829 -lt 25769803829 2024-09-10T17:36:11.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:11.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-313532612616 2024-09-10T17:36:11.564 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:11.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:36:11.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-313532612616 2024-09-10T17:36:11.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:11.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612616 2024-09-10T17:36:11.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 313532612616' 2024-09-10T17:36:11.566 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 313532612616 2024-09-10T17:36:11.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:11.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612614 -lt 313532612616 2024-09-10T17:36:11.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:36:12.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:36:12.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:13.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612616 -lt 313532612616 2024-09-10T17:36:13.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:13.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378662 2024-09-10T17:36:13.380 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:13.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:36:13.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378662 2024-09-10T17:36:13.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:13.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378662 2024-09-10T17:36:13.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-10T17:36:13.382 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378662 2024-09-10T17:36:13.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:36:13.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378662 -lt 81604378662 2024-09-10T17:36:13.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:13.765 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-360777252867 2024-09-10T17:36:13.765 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:13.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:36:13.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-360777252867 2024-09-10T17:36:13.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:13.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252867 2024-09-10T17:36:13.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-09-10T17:36:13.767 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 360777252867 2024-09-10T17:36:13.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:36:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252867 -lt 360777252867 2024-09-10T17:36:14.150 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:36:14.150 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:14.150 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:14.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:36:14.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:36:15.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:36:15.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:36:15.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:15.053 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:15.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:36:15.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:36:15.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:36:15.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:256 bytes copied, 3.3239e-05 s, 7.7 MB/s 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:36:15.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:15.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:15.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:15.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:15.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:36:15.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:36:15.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-10T17:36:15.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:36:15.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-10T17:36:15.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:36:18.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:18.362 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:36:18.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:36:18.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T17:36:18.364 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.3 2024-09-10T17:36:18.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:36:18.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-10T17:36:18.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T17:36:18.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:36:18.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:36:18.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:36:18.392 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:18.391+0000 7f75da02e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:18.394 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:18.393+0000 7f75da02e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:18.396 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:18.395+0000 7f75da02e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:18.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T17:36:18.815 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:18.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:19.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:19.503 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:19.502+0000 7f75da02e640 -1 Falling back to public interface 2024-09-10T17:36:20.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:20.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:20.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:36:20.294 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:20.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:20.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:20.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:21.232 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:21.231+0000 7f75da02e640 -1 osd.3 86 log_to_monitors true 2024-09-10T17:36:21.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:21.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:21.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:36:21.688 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:21.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:21.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:36:22.220 INFO:tasks.workunit.client.0.smithi086.stderr:osd.3 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6826/3401740928,v1:127.0.0.1:6827/3401740928] [v2:127.0.0.1:6828/3401740928,v1:127.0.0.1:6829/3401740928] exists,up 0daa19ca-0796-4d94-960e-1bab306b4758 2024-09-10T17:36:22.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:36:22.221 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:36:22.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:36:22.222 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:36:22.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:36:22.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:36:22.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:36:22.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:36:22.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:36:22.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:36:22.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:36:22.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:36:22.311 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:36:22.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:22.694 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:36:22.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803833 2024-09-10T17:36:22.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803833 2024-09-10T17:36:22.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833' 2024-09-10T17:36:22.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:22.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:36:22.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612620 2024-09-10T17:36:22.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612620 2024-09-10T17:36:22.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-313532612620' 2024-09-10T17:36:22.944 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:22.944 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:36:23.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378666 2024-09-10T17:36:23.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378666 2024-09-10T17:36:23.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-313532612620 2-81604378666' 2024-09-10T17:36:23.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:23.106 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:36:23.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023939 2024-09-10T17:36:23.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023939 2024-09-10T17:36:23.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803833 1-313532612620 2-81604378666 3-390842023939' 2024-09-10T17:36:23.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:23.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803833 2024-09-10T17:36:23.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:23.220 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:36:23.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803833 2024-09-10T17:36:23.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:23.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803833 2024-09-10T17:36:23.221 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-09-10T17:36:23.221 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803833 2024-09-10T17:36:23.222 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:23.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803831 -lt 25769803833 2024-09-10T17:36:23.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:36:24.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:36:24.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:24.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803834 -lt 25769803833 2024-09-10T17:36:24.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:24.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-313532612620 2024-09-10T17:36:24.998 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:24.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:36:24.999 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-313532612620 2024-09-10T17:36:24.999 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:25.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612620 2024-09-10T17:36:25.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 313532612620' 2024-09-10T17:36:25.000 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 313532612620 2024-09-10T17:36:25.000 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:25.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612620 -lt 313532612620 2024-09-10T17:36:25.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:25.380 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378666 2024-09-10T17:36:25.380 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:25.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:36:25.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378666 2024-09-10T17:36:25.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:25.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378666 2024-09-10T17:36:25.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378666' 2024-09-10T17:36:25.382 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378666 2024-09-10T17:36:25.382 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:36:25.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378667 -lt 81604378666 2024-09-10T17:36:25.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:25.761 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-390842023939 2024-09-10T17:36:25.761 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:25.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:36:25.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-390842023939 2024-09-10T17:36:25.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:25.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023939 2024-09-10T17:36:25.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-09-10T17:36:25.764 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 390842023939 2024-09-10T17:36:25.764 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:36:26.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023939 -lt 390842023939 2024-09-10T17:36:26.145 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:36:26.145 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:26.146 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:36:26.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:36:27.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:36:27.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:36:27.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:27.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:27.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:36:27.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:36:27.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:36:27.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:36:27.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:36:28.229 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-256 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:36:28.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-0-256 td/test-erasure-eio/COPY 2024-09-10T17:36:28.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:36:28.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:36:28.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-0-256 td/test-erasure-eio 1 256 add 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-0-256 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:36:28.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-0-256 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-0-256 2024-09-10T17:36:28.443 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:36:28.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:36:28.826 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:28.826 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:36:28.827 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:36:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:36:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:36:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:36:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:36:29.577 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:36:29.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:29.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:29.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:36:29.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:36:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:36:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:36:30.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:36:30.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:30.646 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:36:30.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:36:30.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:36:30.673 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:30.671+0000 7f4b4f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:30.710 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:30.709+0000 7f4b4f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:30.711 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:30.710+0000 7f4b4f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:31.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:36:31.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:36:31.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:31.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:31.571 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:31.570+0000 7f4b4f558640 -1 Falling back to public interface 2024-09-10T17:36:31.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:32.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:32.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:32.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:36:32.612 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:32.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:32.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:33.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:33.839 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:33.837+0000 7f4b4f558640 -1 osd.1 94 log_to_monitors true 2024-09-10T17:36:34.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:34.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:34.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:36:34.020 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:34.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:34.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:34.712 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 101 up_thru 87 down_at 97 last_clean_interval [73,94) [v2:127.0.0.1:6810/3654239922,v1:127.0.0.1:6811/3654239922] [v2:127.0.0.1:6812/3654239922,v1:127.0.0.1:6813/3654239922] exists,up ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:36:34.713 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:36:34.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:36:34.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:36:34.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:36:34.802 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:36:34.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:36:34.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:36:34.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:36:34.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:36:34.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:36:35.499 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:36:35.499 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:35.500 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:35.500 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:36:35.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:36:35.500 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:35.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:36:35.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803838 2024-09-10T17:36:35.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803838 2024-09-10T17:36:35.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838' 2024-09-10T17:36:35.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:35.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:36:35.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696899 2024-09-10T17:36:35.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696899 2024-09-10T17:36:35.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-433791696899' 2024-09-10T17:36:35.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:35.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:36:36.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378671 2024-09-10T17:36:36.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378671 2024-09-10T17:36:36.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-433791696899 2-81604378671' 2024-09-10T17:36:36.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:36.004 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023943 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023943 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803838 1-433791696899 2-81604378671 3-390842023943' 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803838 2024-09-10T17:36:36.115 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:36.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:36:36.116 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803838 2024-09-10T17:36:36.117 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:36.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803838 2024-09-10T17:36:36.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-09-10T17:36:36.118 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803838 2024-09-10T17:36:36.118 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:36.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803836 -lt 25769803838 2024-09-10T17:36:36.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:36:37.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:36:37.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:37.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803838 -lt 25769803838 2024-09-10T17:36:37.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:37.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-433791696899 2024-09-10T17:36:37.923 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:37.924 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:36:37.924 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-433791696899 2024-09-10T17:36:37.924 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:37.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696899 2024-09-10T17:36:37.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 433791696899' 2024-09-10T17:36:37.925 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 433791696899 2024-09-10T17:36:37.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:38.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696899 -lt 433791696899 2024-09-10T17:36:38.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:38.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378671 2024-09-10T17:36:38.351 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:38.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:36:38.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378671 2024-09-10T17:36:38.353 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:38.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378671 2024-09-10T17:36:38.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378671' 2024-09-10T17:36:38.354 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378671 2024-09-10T17:36:38.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:36:38.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378671 -lt 81604378671 2024-09-10T17:36:38.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:38.793 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-390842023943 2024-09-10T17:36:38.793 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:38.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:36:38.794 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-390842023943 2024-09-10T17:36:38.794 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:38.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023943 2024-09-10T17:36:38.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 390842023943' 2024-09-10T17:36:38.796 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 390842023943 2024-09-10T17:36:38.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:36:39.277 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023944 -lt 390842023943 2024-09-10T17:36:39.278 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:36:39.278 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:39.278 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:36:39.833 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:36:39.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:36:40.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:36:40.223 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:36:40.223 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:40.223 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:40.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:36:40.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:36:40.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:36:40.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-10T17:36:40.744 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:256 bytes copied, 3.026e-05 s, 8.5 MB/s 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:36:40.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:40.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:41.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:41.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:41.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:36:41.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:36:41.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:36:41.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:36:41.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:36:41.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:36:43.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:43.908 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:36:43.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:36:43.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:36:43.910 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:36:43.910 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:36:43.911 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:36:43.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:36:43.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:36:43.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:36:43.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:36:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:43.936+0000 7fa2da41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:43.940 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:43.939+0000 7fa2da41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:43.941 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:43.940+0000 7fa2da41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:44.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:36:44.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:44.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:44.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:45.293 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:45.292+0000 7fa2da41c640 -1 Falling back to public interface 2024-09-10T17:36:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:45.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:36:45.745 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:45.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:45.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:46.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:36:46.674 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:46.672+0000 7fa2da41c640 -1 osd.1 103 log_to_monitors true 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:36:47.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [101,103) [v2:127.0.0.1:6810/2001164913,v1:127.0.0.1:6811/2001164913] [v2:127.0.0.1:6812/2001164913,v1:127.0.0.1:6813/2001164913] exists,up ac48c398-4875-4cd0-a24f-24d90a06ea1e 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:36:47.679 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:36:47.680 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:36:47.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:36:47.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:36:47.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:36:47.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:36:47.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:36:47.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:36:47.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:36:47.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:36:47.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:36:48.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:48.414 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:36:48.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803842 2024-09-10T17:36:48.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803842 2024-09-10T17:36:48.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842' 2024-09-10T17:36:48.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:48.549 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:36:48.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467971 2024-09-10T17:36:48.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467971 2024-09-10T17:36:48.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-463856467971' 2024-09-10T17:36:48.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:48.781 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:36:48.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378676 2024-09-10T17:36:48.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378676 2024-09-10T17:36:48.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-463856467971 2-81604378676' 2024-09-10T17:36:48.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:36:48.893 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:36:49.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023948 2024-09-10T17:36:49.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023948 2024-09-10T17:36:49.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-463856467971 2-81604378676 3-390842023948' 2024-09-10T17:36:49.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:49.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803842 2024-09-10T17:36:49.006 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:49.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:36:49.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803842 2024-09-10T17:36:49.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:49.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803842 2024-09-10T17:36:49.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-10T17:36:49.009 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803842 2024-09-10T17:36:49.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:36:49.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-10T17:36:49.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:49.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-463856467971 2024-09-10T17:36:49.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:49.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:36:49.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-463856467971 2024-09-10T17:36:49.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:49.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467971 2024-09-10T17:36:49.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 463856467971' 2024-09-10T17:36:49.391 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 463856467971 2024-09-10T17:36:49.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:49.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 463856467971 2024-09-10T17:36:49.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:36:50.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:36:50.815 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:36:51.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467971 -lt 463856467971 2024-09-10T17:36:51.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:51.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378676 2024-09-10T17:36:51.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:51.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:36:51.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378676 2024-09-10T17:36:51.212 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:51.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378676 2024-09-10T17:36:51.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378676' 2024-09-10T17:36:51.214 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 81604378676 2024-09-10T17:36:51.214 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:36:51.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378676 -lt 81604378676 2024-09-10T17:36:51.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:36:51.599 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-390842023948 2024-09-10T17:36:51.599 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:36:51.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:36:51.601 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-390842023948 2024-09-10T17:36:51.601 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:36:51.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023948 2024-09-10T17:36:51.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 390842023948' 2024-09-10T17:36:51.602 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 390842023948 2024-09-10T17:36:51.602 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:36:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023948 -lt 390842023948 2024-09-10T17:36:52.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:36:52.018 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:52.019 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:36:52.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:36:52.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:36:52.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:36:52.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:36:52.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:36:53.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:36:53.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:36:53.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:36:53.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:36:53.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:36:54.136 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:36:54.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-10T17:36:54.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-0-256 fail 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-0-256 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-10T17:36:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-86744-0-256 td/test-erasure-eio/COPY 2024-09-10T17:36:54.363 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj-size-86744-0-256: (5) Input/output error 2024-09-10T17:36:54.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-10T17:36:54.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:36:54.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:332: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:36:54.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:36:54.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:36:54.964 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:36:54.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:36:55.516 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:36:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:36:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:36:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:36:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:36:55.529 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:55.529 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:55.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:55.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:55.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:55.638 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:55.638 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:36:55.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:36:55.639 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:36:55.640 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:36:55.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:36:55.641 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:36:55.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:36:55.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:36:55.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:36:55.642 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:36:55.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:36:55.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:36:55.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:36:55.672 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:36:55.673 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:55.673 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:55.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:36:55.674 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:36:55.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:36:55.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:36:55.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:36:55.676 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:36:55.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:36:55.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:36:55.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:36:55.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:36:55.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:36:55.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:36:55.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:36:55.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:36:55.680 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:36:55.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:36:55.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:36:55.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:36:55.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:36:55.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:55.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:55.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:36:55.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:36:55.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:36:55.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:36:55.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:36:55.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:55.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:55.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:36:55.687 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:36:55.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:36:55.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:36:55.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:36:56.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:36:56.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:56.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:56.162 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:56.162 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:56.162 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:56.162 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:56.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:36:56.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:36:56.363 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:56.364 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:36:56.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:36:56.364 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:36:56.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:36:56.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:36:56.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:36:56.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:36:56.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:36:57.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:36:57.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:57.365 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:57.365 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:57.366 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:57.366 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:57.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:57.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:36:57.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:36:57.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:36:57.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:36:57.643 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:36:57.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:36:58.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:36:58.727 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:36:58.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:36:58.736 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:36:56.192+0000 7fc128b22c80 0 load: jerasure load: lrc 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:337: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:36:58.737 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:36:58.738 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:36:58.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:36:58.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:36:58.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:36:58.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:36:58.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:36:58.743 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:36:58.744 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ecfab47-bfcd-4151-b10d-d1858ed0dd6d 2024-09-10T17:36:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8ecfab47-bfcd-4151-b10d-d1858ed0dd6d' 2024-09-10T17:36:58.746 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 8ecfab47-bfcd-4151-b10d-d1858ed0dd6d 2024-09-10T17:36:58.746 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:36:58.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6g+BmOg5YLRAAlmIcq4+bF8A7whjsqs1N3A== 2024-09-10T17:36:58.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6g+BmOg5YLRAAlmIcq4+bF8A7whjsqs1N3A=="}' 2024-09-10T17:36:58.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ecfab47-bfcd-4151-b10d-d1858ed0dd6d -i td/test-erasure-eio/0/new.json 2024-09-10T17:36:59.020 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:36:59.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:36:59.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6g+BmOg5YLRAAlmIcq4+bF8A7whjsqs1N3A== --osd-uuid 8ecfab47-bfcd-4151-b10d-d1858ed0dd6d 2024-09-10T17:36:59.060 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:59.058+0000 7f8b9f534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:59.080 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:59.079+0000 7f8b9f534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:59.081 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:59.080+0000 7f8b9f534640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:36:59.082 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:59.080+0000 7f8b9f534640 -1 bdev(0x55abc258bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:36:59.082 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:36:59.081+0000 7f8b9f534640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:37:05.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:37:05.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:37:05.038 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:37:05.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:37:05.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:37:05.566 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:37:05.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:37:05.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:37:05.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:37:05.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:37:05.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:37:05.595 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:05.593+0000 7f3d946e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:05.609 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:05.607+0000 7f3d946e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:05.610 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:05.609+0000 7f3d946e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:06.014 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:06.015 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:37:06.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:37:06.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:06.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:06.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:06.976 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:06.975+0000 7f3d946e0640 -1 Falling back to public interface 2024-09-10T17:37:07.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:07.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:07.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:37:07.442 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:37:07.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:07.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:07.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:08.578 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:08.576+0000 7f3d946e0640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:37:08.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:08.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:08.860 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:37:08.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:37:08.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:08.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:09.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:10.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:10.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:10.334 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:37:10.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:37:10.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:10.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:10.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:11.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:11.743 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:37:11.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:11.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:37:11.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:11.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:12.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:13.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:13.143 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:37:13.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:13.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:37:13.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:13.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:13.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:14.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:14.671 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:37:14.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:14.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:37:14.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:14.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:15.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:16.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:16.067 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:37:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:37:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:16.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:16.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:17.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:17.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:17.480 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:37:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:37:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:17.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:17.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:18.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:18.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:18.878 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:37:18.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:37:18.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:18.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:19.368 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:20.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:20.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:20.370 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:37:20.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:37:20.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:20.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:21.067 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:22.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:22.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:22.069 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:37:22.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:37:22.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:22.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:22.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:23.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:23.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:23.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:37:23.967 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:37:23.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:23.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:24.893 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:25.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:25.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:25.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:37:25.896 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:37:25.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:25.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:26.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:27.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:27.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:27.703 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:37:27.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:37:27.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:27.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:29.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:29.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:29.829 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:37:29.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:37:29.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:29.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:30.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:31.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:31.677 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:37:31.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:31.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:37:31.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:31.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:37:32.148 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2253641357,v1:127.0.0.1:6803/2253641357] [v2:127.0.0.1:6804/2253641357,v1:127.0.0.1:6805/2253641357] exists,up 8ecfab47-bfcd-4151-b10d-d1858ed0dd6d 2024-09-10T17:37:32.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:37:32.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:37:32.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:37:32.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:37:32.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 03b7fe0d-b47a-40c2-bc4e-e534576e21ef' 2024-09-10T17:37:32.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:37:32.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDcg+Bmk5QhChAA0hYE47Eun6rVWFK34SmEHA== 2024-09-10T17:37:32.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDcg+Bmk5QhChAA0hYE47Eun6rVWFK34SmEHA=="}' 2024-09-10T17:37:32.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03b7fe0d-b47a-40c2-bc4e-e534576e21ef -i td/test-erasure-eio/1/new.json 2024-09-10T17:37:32.713 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:37:32.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:37:32.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcg+Bmk5QhChAA0hYE47Eun6rVWFK34SmEHA== --osd-uuid 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:37:32.754 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:32.753+0000 7fac9fce6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:32.755 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:32.754+0000 7fac9fce6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:32.757 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:32.755+0000 7fac9fce6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:32.757 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:32.756+0000 7fac9fce6640 -1 bdev(0x557356a49c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:37:32.757 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:32.756+0000 7fac9fce6640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:37:39.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:37:39.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:37:39.193 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:37:39.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:37:39.194 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:37:39.755 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:37:39.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:37:39.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:37:39.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:37:39.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:37:39.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:37:39.783 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:39.782+0000 7f8c87ba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:39.810 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:39.809+0000 7f8c87ba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:39.811 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:39.810+0000 7f8c87ba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:40.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:37:40.218 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:37:40.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:40.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:40.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:41.163 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:41.162+0000 7f8c87ba8640 -1 Falling back to public interface 2024-09-10T17:37:41.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:41.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:41.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:37:41.635 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:37:41.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:41.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:42.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:42.754 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:37:42.753+0000 7f8c87ba8640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:37:43.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:43.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:43.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:37:43.053 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:37:43.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:43.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:43.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:44.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:44.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:44.683 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:37:44.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:37:44.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:44.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:45.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:46.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:46.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:46.102 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:37:46.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:37:46.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:46.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:46.597 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:47.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:47.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:47.598 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:37:47.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:37:47.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:47.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:48.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:49.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:49.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:49.013 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:37:49.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:37:49.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:49.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:49.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:50.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:50.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:50.417 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:37:50.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:37:50.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:50.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:51.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:51.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:51.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:37:51.918 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:37:51.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:51.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:52.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:53.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:53.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:53.342 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:37:53.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:37:53.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:53.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:53.806 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:54.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:54.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:54.808 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:37:54.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:37:54.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:54.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:55.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:56.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:56.365 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:37:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:37:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:56.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:57.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:37:58.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:37:58.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:37:58.142 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:37:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:37:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:37:58.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:37:59.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:00.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:00.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:00.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:38:00.265 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:38:00.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:00.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:00.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:01.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:01.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:01.772 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:38:01.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:38:01.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:01.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:02.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:03.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:03.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:03.994 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:38:03.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:38:03.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:03.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:04.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:05.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:05.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:05.435 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:38:05.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:38:05.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:05.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:06.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:07.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:07.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:07.124 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:38:07.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:38:07.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:07.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:38:07.659 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/474069404,v1:127.0.0.1:6811/474069404] [v2:127.0.0.1:6812/474069404,v1:127.0.0.1:6813/474069404] exists,up 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:38:07.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:38:07.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:07.664 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:38:07.665 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f78e6385-4165-4818-8aaf-32e479c0d742' 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:38:07.666 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:38:07.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/g+BmjlyJKBAAicpSuFsLYi8D7yYiPg1vzw== 2024-09-10T17:38:07.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/g+BmjlyJKBAAicpSuFsLYi8D7yYiPg1vzw=="}' 2024-09-10T17:38:07.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f78e6385-4165-4818-8aaf-32e479c0d742 -i td/test-erasure-eio/2/new.json 2024-09-10T17:38:08.180 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:38:08.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:38:08.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/g+BmjlyJKBAAicpSuFsLYi8D7yYiPg1vzw== --osd-uuid f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:38:08.222 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:08.221+0000 7fae1d9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:08.224 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:08.223+0000 7fae1d9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:08.225 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:08.224+0000 7fae1d9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:08.225 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:08.224+0000 7fae1d9b0640 -1 bdev(0x55e01ee9bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:38:08.225 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:08.224+0000 7fae1d9b0640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:38:13.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:38:13.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:38:13.767 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:38:13.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:38:13.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:38:14.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:38:14.299 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:38:14.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:38:14.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:38:14.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:38:14.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:38:14.326 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:14.325+0000 7efeed4b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:14.341 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:14.340+0000 7efeed4b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:14.342 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:14.341+0000 7efeed4b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:14.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:14.748 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:38:14.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:38:14.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:14.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:15.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:15.446 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:15.445+0000 7efeed4b6640 -1 Falling back to public interface 2024-09-10T17:38:16.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:16.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:16.174 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:38:16.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:38:16.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:16.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:16.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:17.302 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:17.301+0000 7efeed4b6640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:38:17.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:17.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:17.609 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:38:17.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:38:17.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:17.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:18.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:19.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:19.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:19.130 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:38:19.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:38:19.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:19.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:19.577 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:20.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:20.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:20.578 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:38:20.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:38:20.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:20.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:21.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:22.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:22.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:22.111 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:38:22.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:38:22.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:22.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:23.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:23.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:23.507 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:38:23.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:38:23.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:23.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:23.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:25.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:25.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:25.001 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:38:25.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:38:25.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:25.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:25.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:26.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:26.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:26.398 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:38:26.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:38:26.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:26.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:26.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:27.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:27.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:27.822 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:38:27.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:38:27.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:27.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:28.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:29.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:29.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:29.229 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:38:29.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:38:29.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:29.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:29.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:30.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:30.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:30.627 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:38:30.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:38:30.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:30.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:31.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:32.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:38:32.429 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:38:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:32.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:33.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:34.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:34.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:34.107 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:38:34.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:38:34.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:34.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:35.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:36.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:36.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:36.212 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:38:36.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:38:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:36.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:37.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:37.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:37.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:38:37.981 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:38:37.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:37.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:38.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:39.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:39.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:39.473 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:38:39.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:38:39.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:39.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:39.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:40.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:40.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:40.973 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:38:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:38:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:40.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:38:41.454 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1370549172,v1:127.0.0.1:6819/1370549172] [v2:127.0.0.1:6820/1370549172,v1:127.0.0.1:6821/1370549172] exists,up f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:38:41.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:38:41.459 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:38:41.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e5b698c-9551-48bc-8639-c9300860281d 2024-09-10T17:38:41.461 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 1e5b698c-9551-48bc-8639-c9300860281d 2024-09-10T17:38:41.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1e5b698c-9551-48bc-8639-c9300860281d' 2024-09-10T17:38:41.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:38:41.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhhOBma7hXHBAAz/Bwnc7gg9E7SImqCeddzQ== 2024-09-10T17:38:41.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhhOBma7hXHBAAz/Bwnc7gg9E7SImqCeddzQ=="}' 2024-09-10T17:38:41.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e5b698c-9551-48bc-8639-c9300860281d -i td/test-erasure-eio/3/new.json 2024-09-10T17:38:42.291 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:38:42.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:38:42.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhhOBma7hXHBAAz/Bwnc7gg9E7SImqCeddzQ== --osd-uuid 1e5b698c-9551-48bc-8639-c9300860281d 2024-09-10T17:38:42.333 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:42.332+0000 7f9429c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:42.335 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:42.334+0000 7f9429c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:42.335+0000 7f9429c26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:42.335+0000 7f9429c26640 -1 bdev(0x559004cd7c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:38:42.336 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:42.335+0000 7f9429c26640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:38:49.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:38:49.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:38:49.317 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:38:49.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:38:49.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:38:49.853 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:38:49.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:38:49.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:38:49.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:38:49.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:38:49.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:38:49.880 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:49.879+0000 7fac2d40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:49.909 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:49.907+0000 7fac2d40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:49.910 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:49.909+0000 7fac2d40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:38:50.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:50.273 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:38:50.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:38:50.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:50.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:50.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:51.246 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:51.245+0000 7fac2d40d640 -1 Falling back to public interface 2024-09-10T17:38:51.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:51.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:51.692 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:38:51.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:38:51.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:51.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:52.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:53.063 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:38:53.062+0000 7fac2d40d640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:38:53.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:53.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:53.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:38:53.119 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:38:53.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:53.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:53.737 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:54.738 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:54.739 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:38:54.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:38:54.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:54.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:55.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:56.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:56.406 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:38:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:38:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:56.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:56.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:57.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:57.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:57.928 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:38:57.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:38:57.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:57.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:58.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:38:59.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:38:59.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:38:59.349 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:38:59.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:38:59.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:38:59.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:38:59.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:00.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:00.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:00.840 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:39:00.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:39:00.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:00.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:01.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:02.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:02.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:02.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:39:02.244 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:39:02.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:02.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:02.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:03.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:03.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:03.692 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:39:03.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:39:03.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:03.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:04.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:05.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:05.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:05.149 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:39:05.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:39:05.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:05.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:05.569 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:06.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:06.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:06.571 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:39:06.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:39:06.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:06.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:07.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:08.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:08.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:08.269 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:39:08.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:39:08.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:08.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:08.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:09.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:09.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:09.978 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:39:09.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:39:09.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:09.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:10.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:11.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:11.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:11.963 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:39:11.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:39:11.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:11.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:12.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:13.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:13.652 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:13.652 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:39:13.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:39:13.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:13.653 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:14.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:15.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:15.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:15.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:39:15.212 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:39:15.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:15.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:15.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:16.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:16.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:16.766 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:39:16.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:39:16.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:16.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:17.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:18.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:18.512 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:39:18.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:39:18.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:18.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:39:19.136 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 29 up_thru 31 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3421466838,v1:127.0.0.1:6827/3421466838] [v2:127.0.0.1:6828/3421466838,v1:127.0.0.1:6829/3421466838] exists,up 1e5b698c-9551-48bc-8639-c9300860281d 2024-09-10T17:39:19.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:39:19.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:39:19.218 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:39:19.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:39:19.228 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:37:06.979+0000 7f3d946e0640 0 load: jerasure load: lrc 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:339: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:39:19.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:39:19.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:39:19.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:39:19.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:39:20.941 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:39:20.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:39:21.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:39:21.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:39:21.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:39:21.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:39:21.955 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:39:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:39:21.956 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:39:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:39:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:39:21.956 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:39:22.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:39:22.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:39:22.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:39:22.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:39:22.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:39:22.051 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:39:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:39:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:39:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:39:22.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:39:22.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:22.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:39:22.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803807 2024-09-10T17:39:22.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803807 2024-09-10T17:39:22.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807' 2024-09-10T17:39:22.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:22.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:39:22.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574872 2024-09-10T17:39:22.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574872 2024-09-10T17:39:22.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574872' 2024-09-10T17:39:22.728 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:22.728 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:39:22.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345934 2024-09-10T17:39:22.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345934 2024-09-10T17:39:22.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574872 2-85899345934' 2024-09-10T17:39:22.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:22.840 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:39:22.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051592 2024-09-10T17:39:22.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051592 2024-09-10T17:39:22.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574872 2-85899345934 3-124554051592' 2024-09-10T17:39:22.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:22.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803807 2024-09-10T17:39:22.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:22.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:39:22.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803807 2024-09-10T17:39:22.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:22.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803807 2024-09-10T17:39:22.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-10T17:39:22.953 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803807 2024-09-10T17:39:22.954 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803804 -lt 25769803807 2024-09-10T17:39:23.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:39:24.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:39:24.369 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:24.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803807 -lt 25769803807 2024-09-10T17:39:24.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:24.754 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574872 2024-09-10T17:39:24.755 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:24.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:39:24.756 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574872 2024-09-10T17:39:24.756 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:24.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574872 2024-09-10T17:39:24.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-09-10T17:39:24.757 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574872 2024-09-10T17:39:24.757 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:39:25.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574872 -lt 55834574872 2024-09-10T17:39:25.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:25.141 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345934 2024-09-10T17:39:25.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:25.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:39:25.142 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345934 2024-09-10T17:39:25.144 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:25.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345934 2024-09-10T17:39:25.145 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-09-10T17:39:25.145 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 85899345934 2024-09-10T17:39:25.145 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:39:25.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345934 -lt 85899345934 2024-09-10T17:39:25.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:25.529 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051592 2024-09-10T17:39:25.529 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:25.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:39:25.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051592 2024-09-10T17:39:25.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:25.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051592 2024-09-10T17:39:25.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051592' 2024-09-10T17:39:25.532 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 124554051592 2024-09-10T17:39:25.532 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:39:25.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051593 -lt 124554051592 2024-09-10T17:39:25.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:39:25.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:25.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:26.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:39:26.434 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:39:26.434 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:39:26.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:39:26.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:39:26.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:39:26.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:39:26.435 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:39:26.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:39:26.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:39:26.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:26.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:342: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:343: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-10T17:39:27.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-1-10 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-1-10 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:39:27.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:39:27.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-1-10 td/test-erasure-eio/ORIGINAL 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-1-10 td/test-erasure-eio 1 10 set 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-10 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:39:27.534 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-10 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:27.535 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:39:27.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:39:28.568 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:39:28.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:39:28.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-10T17:39:28.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-10T17:39:28.581 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:10 bytes copied, 4.4763e-05 s, 223 kB/s 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:39:28.582 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:28.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:28.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:28.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:28.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:39:29.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:39:31.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:39:31.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:39:31.788 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:39:31.789 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:39:31.816 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:31.815+0000 7fba5dc19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:31.819 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:31.817+0000 7fba5dc19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:31.820 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:31.819+0000 7fba5dc19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:39:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:39:32.596 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:33.410 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:33.409+0000 7fba5dc19640 -1 Falling back to public interface 2024-09-10T17:39:33.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:33.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:33.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:39:33.600 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:33.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:33.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:39:34.023 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:35.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:39:35.080 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:35.079+0000 7fba5dc19640 -1 osd.1 37 log_to_monitors true 2024-09-10T17:39:35.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:36.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:36.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:36.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:39:36.417 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:39:36.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:36.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [13,37) [v2:127.0.0.1:6810/1897132939,v1:127.0.0.1:6811/1897132939] [v2:127.0.0.1:6812/1897132939,v1:127.0.0.1:6813/1897132939] exists,up 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:39:36.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:39:36.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:39:36.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:39:36.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:39:36.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:39:36.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:39:36.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:39:36.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:39:37.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:37.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:39:37.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-10T17:39:37.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-10T17:39:37.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-10T17:39:37.489 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:37.489 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:39:37.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593731 2024-09-10T17:39:37.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593731 2024-09-10T17:39:37.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-184683593731' 2024-09-10T17:39:37.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:37.625 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:39:37.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345939 2024-09-10T17:39:37.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345939 2024-09-10T17:39:37.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-184683593731 2-85899345939' 2024-09-10T17:39:37.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:37.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:39:37.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051597 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051597 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-184683593731 2-85899345939 3-124554051597' 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:37.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803812 2024-09-10T17:39:37.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:38.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803810 -lt 25769803812 2024-09-10T17:39:38.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:39:39.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:39:39.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:39.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-10T17:39:39.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:39.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593731 2024-09-10T17:39:39.659 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:39.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:39:39.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593731 2024-09-10T17:39:39.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593731 2024-09-10T17:39:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-09-10T17:39:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 184683593731 2024-09-10T17:39:39.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:39:40.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593731 -lt 184683593731 2024-09-10T17:39:40.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:40.046 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345939 2024-09-10T17:39:40.046 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:40.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:39:40.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345939 2024-09-10T17:39:40.047 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:40.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345939 2024-09-10T17:39:40.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-10T17:39:40.049 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 85899345939 2024-09-10T17:39:40.049 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:39:40.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-10T17:39:40.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:40.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051597 2024-09-10T17:39:40.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:40.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:39:40.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051597 2024-09-10T17:39:40.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:40.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051597 2024-09-10T17:39:40.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051597' 2024-09-10T17:39:40.474 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051597 2024-09-10T17:39:40.475 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:39:40.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051598 -lt 124554051597 2024-09-10T17:39:40.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:39:40.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:40.854 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:41.369 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:39:41.370 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:39:41.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:39:41.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:39:41.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:41.744 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:39:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:39:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:39:42.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:39:42.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:39:42.982 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-10 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:39:42.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-10 td/test-erasure-eio/COPY 2024-09-10T17:39:43.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:39:43.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:39:43.154 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-10T17:39:43.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-10T17:39:43.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-1-10 td/test-erasure-eio 2 10 set 2024-09-10T17:39:43.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-10 2024-09-10T17:39:43.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:39:43.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:43.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:39:43.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-10 2024-09-10T17:39:43.159 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-10 2024-09-10T17:39:43.159 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:39:43.573 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-10T17:39:43.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:39:44.245 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:39:44.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:39:44.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2024-09-10T17:39:44.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-10T17:39:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:39:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:39:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:10 bytes copied, 4.384e-05 s, 228 kB/s 2024-09-10T17:39:44.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:39:44.260 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:39:44.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:39:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:39:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:39:44.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-10T17:39:44.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-86744-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:39:47.197 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:39:47.198 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:39:47.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:39:47.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:39:47.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.2 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:39:47.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:39:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:47.228+0000 7fbdfe42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:47.230+0000 7fbdfe42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:47.233 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:47.231+0000 7fbdfe42f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:39:47.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:39:47.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:39:47.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:39:47.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:39:47.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:39:47.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:47.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:39:47.617 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:39:47.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:47.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:39:48.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:48.086 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:48.085+0000 7fbdfe42f640 -1 Falling back to public interface 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:39:49.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:39:50.219 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:39:50.218+0000 7fbdfe42f640 -1 osd.2 48 log_to_monitors true 2024-09-10T17:39:50.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:39:50.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:39:50.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:39:50.527 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:39:50.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:39:50.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 up in weight 1 up_from 54 up_thru 40 down_at 50 last_clean_interval [20,48) [v2:127.0.0.1:6818/704070377,v1:127.0.0.1:6819/704070377] [v2:127.0.0.1:6820/704070377,v1:127.0.0.1:6821/704070377] exists,up f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:39:51.193 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:39:51.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:39:51.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:39:51.194 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:39:51.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:39:51.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:39:51.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:39:51.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:39:51.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:39:51.283 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:39:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:39:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:39:51.779 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:39:51.780 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:39:51.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:39:51.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:51.780 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:39:51.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803816 2024-09-10T17:39:51.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803816 2024-09-10T17:39:51.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816' 2024-09-10T17:39:51.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:51.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:39:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593736 2024-09-10T17:39:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593736 2024-09-10T17:39:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-184683593736' 2024-09-10T17:39:52.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:52.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:39:52.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233987 2024-09-10T17:39:52.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233987 2024-09-10T17:39:52.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-184683593736 2-231928233987' 2024-09-10T17:39:52.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:39:52.202 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:39:52.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051602 2024-09-10T17:39:52.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051602 2024-09-10T17:39:52.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803816 1-184683593736 2-231928233987 3-124554051602' 2024-09-10T17:39:52.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:52.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803816 2024-09-10T17:39:52.317 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:52.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:39:52.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803816 2024-09-10T17:39:52.318 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:52.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803816 2024-09-10T17:39:52.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-10T17:39:52.319 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803816 2024-09-10T17:39:52.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:52.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803814 -lt 25769803816 2024-09-10T17:39:52.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:39:53.700 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:39:53.700 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:39:54.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803817 -lt 25769803816 2024-09-10T17:39:54.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:54.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593736 2024-09-10T17:39:54.121 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:54.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:39:54.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593736 2024-09-10T17:39:54.122 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:54.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593736 2024-09-10T17:39:54.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593736' 2024-09-10T17:39:54.123 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 184683593736 2024-09-10T17:39:54.124 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:39:54.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593736 -lt 184683593736 2024-09-10T17:39:54.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:54.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-231928233987 2024-09-10T17:39:54.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:54.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:39:54.540 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-231928233987 2024-09-10T17:39:54.540 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:54.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233987 2024-09-10T17:39:54.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 231928233987' 2024-09-10T17:39:54.541 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 231928233987 2024-09-10T17:39:54.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:39:54.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233987 -lt 231928233987 2024-09-10T17:39:54.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:39:54.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051602 2024-09-10T17:39:54.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:39:54.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:39:54.966 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051602 2024-09-10T17:39:54.967 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:39:54.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051602 2024-09-10T17:39:54.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051602' 2024-09-10T17:39:54.967 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051602 2024-09-10T17:39:54.968 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:39:55.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051602 -lt 124554051602 2024-09-10T17:39:55.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:39:55.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:55.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:55.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:39:55.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:39:56.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T17:39:56.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:39:56.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:56.316 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:39:56.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:39:56.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:39:56.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:39:56.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:39:56.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:39:56.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:39:56.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:39:56.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:39:56.937 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:39:57.309 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T17:39:57.310 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:39:57.310 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:57.310 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T17:39:57.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T17:39:58.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T17:39:58.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T17:39:58.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T17:39:58.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:39:58.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T17:39:58.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:39:58.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:39:58.588 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:39:58.588 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:39:58.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:39:58.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:39:58.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:39:58.589 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:39:58.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:39:58.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:39:58.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:39:58.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:39:59.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:39:59.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:39:59.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:39:59.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:39:59.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:40:00.128 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:40:00.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-10T17:40:00.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-10 2024-09-10T17:40:00.141 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:40:00.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:40:00.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-10 2024-09-10T17:40:00.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:40:00.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:40:00.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-10 td/test-erasure-eio/COPY 2024-09-10T17:40:00.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:40:00.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:40:00.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:40:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-1-0 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-1-0 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:00.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:40:00.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:00.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:40:00.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-1-0 td/test-erasure-eio/ORIGINAL 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-1-0 td/test-erasure-eio 1 0 set 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-0 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:40:00.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-0 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:00.452 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:40:00.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:40:00.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:40:01.602 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:40:01.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:40:01.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-10T17:40:01.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-10T17:40:01.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:01.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:01.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:01.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:01.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:40:01.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:01.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:40:04.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:04.646 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:04.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:40:04.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:40:04.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:40:04.649 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:40:04.677 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:04.676+0000 7f73e6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:04.679 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:04.678+0000 7f73e6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:04.680 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:04.679+0000 7f73e6d58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:05.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:05.073 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:40:05.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:05.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:05.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:06.270 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:06.269+0000 7f73e6d58640 -1 Falling back to public interface 2024-09-10T17:40:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:06.480 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:06.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:06.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:06.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:07.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:07.971 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:07.970+0000 7f73e6d58640 -1 osd.1 58 log_to_monitors true 2024-09-10T17:40:08.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:09.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:09.858 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 61 last_clean_interval [43,58) [v2:127.0.0.1:6810/666265331,v1:127.0.0.1:6811/666265331] [v2:127.0.0.1:6812/666265331,v1:127.0.0.1:6813/666265331] exists,up 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:40:09.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:09.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:09.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:09.858 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:40:09.859 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:40:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:40:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:40:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:40:09.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:40:09.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:40:09.949 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:40:10.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:40:10.335 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:10.336 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:10.336 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:40:10.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:40:10.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:10.336 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:40:10.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803822 2024-09-10T17:40:10.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803822 2024-09-10T17:40:10.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822' 2024-09-10T17:40:10.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:10.460 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:40:10.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874243 2024-09-10T17:40:10.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874243 2024-09-10T17:40:10.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-279172874243' 2024-09-10T17:40:10.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:10.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:40:10.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233993 2024-09-10T17:40:10.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233993 2024-09-10T17:40:10.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-279172874243 2-231928233993' 2024-09-10T17:40:10.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:10.731 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:40:10.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051608 2024-09-10T17:40:10.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051608 2024-09-10T17:40:10.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803822 1-279172874243 2-231928233993 3-124554051608' 2024-09-10T17:40:10.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:10.842 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803822 2024-09-10T17:40:10.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:10.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:40:10.844 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803822 2024-09-10T17:40:10.844 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:10.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803822 2024-09-10T17:40:10.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-10T17:40:10.845 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803822 2024-09-10T17:40:10.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:11.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803820 -lt 25769803822 2024-09-10T17:40:11.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:40:12.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:40:12.297 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:12.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-10T17:40:12.723 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:12.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-279172874243 2024-09-10T17:40:12.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:12.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:40:12.725 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-279172874243 2024-09-10T17:40:12.725 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:12.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874243 2024-09-10T17:40:12.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2024-09-10T17:40:12.726 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 279172874243 2024-09-10T17:40:12.726 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:40:13.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874243 -lt 279172874243 2024-09-10T17:40:13.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:13.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-231928233993 2024-09-10T17:40:13.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:13.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:40:13.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-231928233993 2024-09-10T17:40:13.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:13.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233993 2024-09-10T17:40:13.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 231928233993' 2024-09-10T17:40:13.160 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 231928233993 2024-09-10T17:40:13.160 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:40:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233993 -lt 231928233993 2024-09-10T17:40:13.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:13.542 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051608 2024-09-10T17:40:13.542 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:13.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:40:13.543 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051608 2024-09-10T17:40:13.543 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:13.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051608 2024-09-10T17:40:13.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051608' 2024-09-10T17:40:13.545 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051608 2024-09-10T17:40:13.545 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:40:13.943 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051608 -lt 124554051608 2024-09-10T17:40:13.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:40:13.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:13.943 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:14.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:40:14.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:14.470 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:14.470 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:14.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:14.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:14.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:14.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:14.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:40:14.869 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:14.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:14.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:15.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:40:15.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:40:15.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:40:15.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:40:15.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:40:16.135 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-0 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:40:16.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-0 td/test-erasure-eio/COPY 2024-09-10T17:40:16.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:40:16.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:40:16.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-1-0 td/test-erasure-eio 2 0 set 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-0 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2024-09-10T17:40:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-0 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:40:16.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:40:16.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-10T17:40:16.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:40:17.348 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:40:17.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2024-09-10T17:40:17.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2024-09-10T17:40:17.360 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-10T17:40:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:40:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:40:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T17:40:17.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:17.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:40:18.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-10T17:40:18.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-86744-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:20.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:20.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:20.823 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:20.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.2 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:40:20.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:40:20.850 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:20.849+0000 7f3fb2e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:20.852 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:20.851+0000 7f3fb2e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:20.853 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:20.852+0000 7f3fb2e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:21.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:40:21.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:21.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:40:21.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:22.205 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:22.204+0000 7f3fb2e0b640 -1 Falling back to public interface 2024-09-10T17:40:22.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:22.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:22.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:22.707 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:22.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:22.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:40:23.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:23.907 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:23.906+0000 7f3fb2e0b640 -1 osd.2 72 log_to_monitors true 2024-09-10T17:40:24.102 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:24.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:24.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:24.103 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:24.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:24.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 up in weight 1 up_from 77 up_thru 61 down_at 73 last_clean_interval [54,72) [v2:127.0.0.1:6818/3933567612,v1:127.0.0.1:6819/3933567612] [v2:127.0.0.1:6820/3933567612,v1:127.0.0.1:6821/3933567612] exists,up f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:40:24.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:40:24.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:40:24.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:40:24.705 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:40:24.705 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:40:24.705 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:40:24.706 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:40:24.706 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:40:24.706 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:40:24.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:40:24.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:40:24.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:40:24.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:40:24.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:40:24.795 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:25.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:40:25.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803827 2024-09-10T17:40:25.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803827 2024-09-10T17:40:25.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827' 2024-09-10T17:40:25.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:25.612 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:40:25.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874248 2024-09-10T17:40:25.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874248 2024-09-10T17:40:25.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-279172874248' 2024-09-10T17:40:25.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:25.734 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:40:25.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481795 2024-09-10T17:40:25.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481795 2024-09-10T17:40:25.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-279172874248 2-330712481795' 2024-09-10T17:40:25.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:25.890 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:40:26.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051613 2024-09-10T17:40:26.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051613 2024-09-10T17:40:26.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803827 1-279172874248 2-330712481795 3-124554051613' 2024-09-10T17:40:26.088 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:26.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803827 2024-09-10T17:40:26.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:26.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:40:26.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803827 2024-09-10T17:40:26.090 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:26.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803827 2024-09-10T17:40:26.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-10T17:40:26.091 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803827 2024-09-10T17:40:26.091 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:26.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803825 -lt 25769803827 2024-09-10T17:40:26.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:40:27.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:40:27.509 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:27.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803828 -lt 25769803827 2024-09-10T17:40:27.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:27.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-279172874248 2024-09-10T17:40:27.935 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:27.935 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:40:27.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-279172874248 2024-09-10T17:40:27.936 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:27.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874248 2024-09-10T17:40:27.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 279172874248' 2024-09-10T17:40:27.937 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 279172874248 2024-09-10T17:40:27.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:40:28.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874248 -lt 279172874248 2024-09-10T17:40:28.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:28.352 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-330712481795 2024-09-10T17:40:28.353 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:28.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:40:28.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-330712481795 2024-09-10T17:40:28.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:28.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481795 2024-09-10T17:40:28.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 330712481795' 2024-09-10T17:40:28.355 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 330712481795 2024-09-10T17:40:28.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:40:28.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481795 -lt 330712481795 2024-09-10T17:40:28.739 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:28.739 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051613 2024-09-10T17:40:28.739 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:28.740 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:40:28.740 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051613 2024-09-10T17:40:28.741 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:28.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051613 2024-09-10T17:40:28.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051613' 2024-09-10T17:40:28.742 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051613 2024-09-10T17:40:28.742 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:40:29.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051613 -lt 124554051613 2024-09-10T17:40:29.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:40:29.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:29.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:29.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:29.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:29.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:30.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:40:30.064 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:30.064 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:30.064 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:40:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:40:30.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:40:30.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:40:30.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:40:31.212 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:40:31.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-10T17:40:31.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-0 2024-09-10T17:40:31.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:40:31.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:40:31.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-0 2024-09-10T17:40:31.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:40:31.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:40:31.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-0 td/test-erasure-eio/COPY 2024-09-10T17:40:31.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:40:31.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:40:31.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2024-09-10T17:40:31.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-86744-1-256 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-86744-1-256 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-86744-1-256 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:40:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:40:31.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-86744-1-256 td/test-erasure-eio/ORIGINAL 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-86744-1-256 td/test-erasure-eio 1 256 add 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-256 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:40:31.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-256 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-256 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-256 2024-09-10T17:40:31.519 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:40:31.912 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2024-09-10T17:40:31.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:40:32.527 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:32.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:32.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:32.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:32.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:32.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:33.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:33.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:33.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:40:33.663 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:40:33.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:40:33.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:40:33.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:40:33.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:40:33.687 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:33.686+0000 7f8b87afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:33.762 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:33.760+0000 7f8b87afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:33.762 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:33.761+0000 7f8b87afe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:34.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:34.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:34.133 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:40:34.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:34.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:34.576 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:34.855 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:34.854+0000 7f8b87afe640 -1 Falling back to public interface 2024-09-10T17:40:35.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:35.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:35.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:35.578 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:35.578 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:35.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:35.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:36.889 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:36.888+0000 7f8b87afe640 -1 osd.1 81 log_to_monitors true 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:36.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:37.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:38.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:38.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:38.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:40:38.435 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:40:38.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:38.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 84 last_clean_interval [65,81) [v2:127.0.0.1:6810/2438653512,v1:127.0.0.1:6811/2438653512] [v2:127.0.0.1:6812/2438653512,v1:127.0.0.1:6813/2438653512] exists,up 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:40:38.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:40:38.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:40:38.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:40:38.850 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:40:38.850 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:40:38.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:40:38.851 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:40:38.851 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:40:38.940 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:40:38.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:40:38.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:40:38.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:40:38.941 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:40:38.941 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:39.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:40:39.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803832 2024-09-10T17:40:39.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803832 2024-09-10T17:40:39.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832' 2024-09-10T17:40:39.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:39.511 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:40:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=373662154755 2024-09-10T17:40:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 373662154755 2024-09-10T17:40:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-373662154755' 2024-09-10T17:40:39.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:39.647 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:40:39.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481800 2024-09-10T17:40:39.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481800 2024-09-10T17:40:39.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-373662154755 2-330712481800' 2024-09-10T17:40:39.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:39.758 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051618 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051618 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803832 1-373662154755 2-330712481800 3-124554051618' 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803832 2024-09-10T17:40:39.869 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:39.870 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:40:39.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803832 2024-09-10T17:40:39.871 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:39.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803832 2024-09-10T17:40:39.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-10T17:40:39.872 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803832 2024-09-10T17:40:39.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:40.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803830 -lt 25769803832 2024-09-10T17:40:40.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:40:41.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:40:41.291 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:41.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-10T17:40:41.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:41.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-373662154755 2024-09-10T17:40:41.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:41.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:40:41.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-373662154755 2024-09-10T17:40:41.685 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:41.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=373662154755 2024-09-10T17:40:41.686 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 373662154755' 2024-09-10T17:40:41.686 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 373662154755 2024-09-10T17:40:41.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:40:42.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 373662154755 -lt 373662154755 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-330712481800 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-330712481800 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481800 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 330712481800' 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 330712481800 2024-09-10T17:40:42.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:40:42.535 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481800 -lt 330712481800 2024-09-10T17:40:42.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:42.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051618 2024-09-10T17:40:42.536 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:42.536 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:40:42.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051618 2024-09-10T17:40:42.537 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:42.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051618 2024-09-10T17:40:42.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051618' 2024-09-10T17:40:42.538 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051618 2024-09-10T17:40:42.539 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:40:42.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051618 -lt 124554051618 2024-09-10T17:40:42.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:40:42.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:42.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:43.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:40:43.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:43.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:43.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:43.482 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:43.482 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:43.482 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:43.482 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:43.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:40:43.891 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:43.891 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:43.891 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:44.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:40:44.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:40:44.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:40:44.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:256 bytes copied, 3.2986e-05 s, 7.8 MB/s 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:40:44.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:40:44.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:40:44.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:44.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:40:47.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:40:47.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:40:47.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:40:47.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:40:47.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:40:47.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:40:47.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:40:47.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:40:47.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:40:47.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:40:47.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:40:47.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:40:47.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:40:47.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:40:47.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:40:47.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:40:47.256 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:47.254+0000 7f2ed2cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:47.258 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:47.257+0000 7f2ed2cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:47.260 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:47.259+0000 7f2ed2cf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:40:47.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:40:47.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:47.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:48.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:48.617 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:48.616+0000 7f2ed2cf1640 -1 Falling back to public interface 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:49.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:49.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:40:50.399 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:40:50.398+0000 7f2ed2cf1640 -1 osd.1 89 log_to_monitors true 2024-09-10T17:40:50.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:40:50.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:40:50.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:40:50.474 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:50.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:40:50.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 94 up_thru 87 down_at 90 last_clean_interval [87,89) [v2:127.0.0.1:6810/2573529621,v1:127.0.0.1:6811/2573529621] [v2:127.0.0.1:6812/2573529621,v1:127.0.0.1:6813/2573529621] exists,up 03b7fe0d-b47a-40c2-bc4e-e534576e21ef 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:40:50.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:40:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:40:50.920 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:40:50.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:40:50.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:40:50.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:40:51.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:40:51.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:40:51.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:40:51.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:40:51.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:40:51.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:40:51.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:51.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:40:51.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803836 2024-09-10T17:40:51.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803836 2024-09-10T17:40:51.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836' 2024-09-10T17:40:51.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:51.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:40:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=403726925827 2024-09-10T17:40:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 403726925827 2024-09-10T17:40:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-403726925827' 2024-09-10T17:40:52.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:52.019 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:40:52.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481804 2024-09-10T17:40:52.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481804 2024-09-10T17:40:52.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-403726925827 2-330712481804' 2024-09-10T17:40:52.144 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:40:52.144 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051622 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051622 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803836 1-403726925827 2-330712481804 3-124554051622' 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803836 2024-09-10T17:40:52.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:52.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:40:52.304 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803836 2024-09-10T17:40:52.304 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:52.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803836 2024-09-10T17:40:52.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-10T17:40:52.305 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803836 2024-09-10T17:40:52.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:52.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803834 -lt 25769803836 2024-09-10T17:40:52.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:40:53.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:40:53.716 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:40:54.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803837 -lt 25769803836 2024-09-10T17:40:54.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:54.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-403726925827 2024-09-10T17:40:54.133 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:54.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:40:54.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-403726925827 2024-09-10T17:40:54.134 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:54.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=403726925827 2024-09-10T17:40:54.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-09-10T17:40:54.135 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 403726925827 2024-09-10T17:40:54.136 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:40:54.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 403726925827 -lt 403726925827 2024-09-10T17:40:54.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:54.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-330712481804 2024-09-10T17:40:54.568 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:54.568 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:40:54.569 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-330712481804 2024-09-10T17:40:54.569 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:54.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481804 2024-09-10T17:40:54.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 330712481804' 2024-09-10T17:40:54.570 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 330712481804 2024-09-10T17:40:54.571 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:40:54.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481804 -lt 330712481804 2024-09-10T17:40:54.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:40:54.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051622 2024-09-10T17:40:54.994 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:40:54.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:40:54.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051622 2024-09-10T17:40:54.995 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:40:54.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051622 2024-09-10T17:40:54.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051622' 2024-09-10T17:40:54.996 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051622 2024-09-10T17:40:54.997 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:40:55.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051622 -lt 124554051622 2024-09-10T17:40:55.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:40:55.421 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:55.421 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:55.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:40:55.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:55.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:55.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:55.938 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:55.939 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:55.939 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:55.939 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:56.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:40:56.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:56.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:56.341 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:40:56.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T17:40:56.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:40:56.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:56.969 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:57.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-10T17:40:57.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:57.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:57.342 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T17:40:57.867 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T17:40:57.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T17:40:58.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T17:40:58.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T17:40:58.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-10T17:40:58.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T17:40:58.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T17:40:58.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T17:40:58.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:40:58.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:40:58.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:40:58.615 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:40:58.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:40:58.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:40:58.616 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:40:58.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:40:58.988 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:40:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:40:58.989 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:40:59.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:40:59.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:40:59.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:40:59.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:40:59.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:41:00.141 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:41:00.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-256 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-256 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:41:00.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-256 td/test-erasure-eio/COPY 2024-09-10T17:41:00.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:41:00.330 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:41:00.331 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2024-09-10T17:41:00.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2024-09-10T17:41:00.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-86744-1-256 td/test-erasure-eio 2 256 add 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-86744-1-256 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2024-09-10T17:41:00.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-86744-1-256 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:41:00.337 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-86744-1-256 2024-09-10T17:41:00.338 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-86744-1-256 2024-09-10T17:41:00.338 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2024-09-10T17:41:00.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2024-09-10T17:41:01.387 INFO:tasks.workunit.client.0.smithi086.stderr:noout is set 2024-09-10T17:41:01.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2024-09-10T17:41:01.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:01.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:41:01.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:01.401 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:01.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:01.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:01.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:01.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-10T17:41:01.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-86744-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:41:02.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:02.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:41:02.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:41:02.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:41:02.455 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.2 2024-09-10T17:41:02.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:41:02.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:41:02.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:41:02.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:41:02.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:41:02.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:41:02.479 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:02.477+0000 7f13cc40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:02.495 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:02.493+0000 7f13cc40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:02.496 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:02.495+0000 7f13cc40b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:02.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:41:02.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:41:02.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:41:02.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:02.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:03.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:03.587 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:03.586+0000 7f13cc40b640 -1 Falling back to public interface 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:04.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:04.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:05.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:05.949 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:05.947+0000 7f13cc40b640 -1 osd.2 96 log_to_monitors true 2024-09-10T17:41:06.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:07.307 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:07.831 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 up in weight 1 up_from 104 up_thru 90 down_at 101 last_clean_interval [77,96) [v2:127.0.0.1:6818/813325997,v1:127.0.0.1:6819/813325997] [v2:127.0.0.1:6820/813325997,v1:127.0.0.1:6821/813325997] exists,up f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:41:07.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:41:07.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:41:07.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:41:07.832 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:41:07.921 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:41:08.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:41:08.417 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:41:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:41:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:41:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:41:08.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:08.418 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:41:08.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803842 2024-09-10T17:41:08.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803842 2024-09-10T17:41:08.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842' 2024-09-10T17:41:08.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:08.530 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:41:08.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=403726925832 2024-09-10T17:41:08.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 403726925832 2024-09-10T17:41:08.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-403726925832' 2024-09-10T17:41:08.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:08.642 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:41:08.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598787 2024-09-10T17:41:08.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598787 2024-09-10T17:41:08.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-403726925832 2-446676598787' 2024-09-10T17:41:08.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:08.753 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051628 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051628 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803842 1-403726925832 2-446676598787 3-124554051628' 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803842 2024-09-10T17:41:08.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:08.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:41:08.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803842 2024-09-10T17:41:08.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:08.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803842 2024-09-10T17:41:08.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-10T17:41:08.868 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803842 2024-09-10T17:41:08.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:09.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803840 -lt 25769803842 2024-09-10T17:41:09.246 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:41:10.247 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:41:10.247 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:10.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-10T17:41:10.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:10.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-403726925832 2024-09-10T17:41:10.679 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:10.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:41:10.681 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-403726925832 2024-09-10T17:41:10.681 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:10.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=403726925832 2024-09-10T17:41:10.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 403726925832' 2024-09-10T17:41:10.683 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 403726925832 2024-09-10T17:41:10.683 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:41:11.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 403726925832 -lt 403726925832 2024-09-10T17:41:11.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:11.105 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-446676598787 2024-09-10T17:41:11.106 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:11.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:41:11.107 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-446676598787 2024-09-10T17:41:11.107 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:11.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598787 2024-09-10T17:41:11.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 446676598787' 2024-09-10T17:41:11.108 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 446676598787 2024-09-10T17:41:11.108 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:41:11.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598788 -lt 446676598787 2024-09-10T17:41:11.490 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:11.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051628 2024-09-10T17:41:11.491 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:11.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:41:11.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051628 2024-09-10T17:41:11.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:11.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051628 2024-09-10T17:41:11.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051628' 2024-09-10T17:41:11.493 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051628 2024-09-10T17:41:11.493 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:41:11.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051628 -lt 124554051628 2024-09-10T17:41:11.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:41:11.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:41:11.925 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:41:12.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:41:12.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:41:12.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:41:12.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:41:12.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:41:12.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:41:12.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:41:12.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:41:12.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:41:12.844 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:41:12.844 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:41:12.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:41:13.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:41:13.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:41:13.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:41:13.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records in 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:1+0 records out 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:256 bytes copied, 3.1558e-05 s, 8.1 MB/s 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:41:13.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:13.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-10T17:41:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-86744-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:41:16.355 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:16.356 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:41:16.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.2 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:41:16.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:41:16.386 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:16.385+0000 7f2c7d325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:16.388 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:16.387+0000 7f2c7d325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:16.390 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:16.388+0000 7f2c7d325640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:41:16.786 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:41:16.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:16.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:17.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:17.243 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:17.242+0000 7f2c7d325640 -1 Falling back to public interface 2024-09-10T17:41:18.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:18.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:18.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:41:18.173 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:41:18.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:18.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:18.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:19.088 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:19.086+0000 7f2c7d325640 -1 osd.2 106 log_to_monitors true 2024-09-10T17:41:19.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:19.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:19.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:41:19.624 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:41:19.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:19.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:41:20.168 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 up in weight 1 up_from 111 up_thru 90 down_at 107 last_clean_interval [104,106) [v2:127.0.0.1:6818/3679594355,v1:127.0.0.1:6819/3679594355] [v2:127.0.0.1:6820/3679594355,v1:127.0.0.1:6821/3679594355] exists,up f78e6385-4165-4818-8aaf-32e479c0d742 2024-09-10T17:41:20.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:41:20.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:41:20.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:41:20.168 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:41:20.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:41:20.258 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:20.861 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:41:21.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803846 2024-09-10T17:41:21.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803846 2024-09-10T17:41:21.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846' 2024-09-10T17:41:21.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:21.031 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:41:21.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=403726925837 2024-09-10T17:41:21.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 403726925837 2024-09-10T17:41:21.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-403726925837' 2024-09-10T17:41:21.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:21.167 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:41:21.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=476741369859 2024-09-10T17:41:21.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 476741369859 2024-09-10T17:41:21.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-403726925837 2-476741369859' 2024-09-10T17:41:21.279 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:41:21.279 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051632 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051632 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803846 1-403726925837 2-476741369859 3-124554051632' 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803846 2024-09-10T17:41:21.391 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:21.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:41:21.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803846 2024-09-10T17:41:21.392 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:21.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803846 2024-09-10T17:41:21.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-10T17:41:21.394 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803846 2024-09-10T17:41:21.394 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:41:21.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-10T17:41:21.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:21.773 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-403726925837 2024-09-10T17:41:21.773 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:21.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:41:21.774 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-403726925837 2024-09-10T17:41:21.774 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:21.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=403726925837 2024-09-10T17:41:21.776 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 403726925837' 2024-09-10T17:41:21.776 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 403726925837 2024-09-10T17:41:21.776 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:41:22.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 403726925837 -lt 403726925837 2024-09-10T17:41:22.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:22.180 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-476741369859 2024-09-10T17:41:22.180 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:22.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:41:22.181 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-476741369859 2024-09-10T17:41:22.182 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:22.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=476741369859 2024-09-10T17:41:22.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 476741369859' 2024-09-10T17:41:22.183 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 476741369859 2024-09-10T17:41:22.183 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:41:22.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 476741369859 -lt 476741369859 2024-09-10T17:41:22.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:41:22.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-124554051632 2024-09-10T17:41:22.607 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:41:22.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:41:22.608 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-124554051632 2024-09-10T17:41:22.608 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:41:22.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051632 2024-09-10T17:41:22.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 124554051632' 2024-09-10T17:41:22.609 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 124554051632 2024-09-10T17:41:22.610 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:41:22.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051630 -lt 124554051632 2024-09-10T17:41:22.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:41:23.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:41:23.996 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:41:24.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051632 -lt 124554051632 2024-09-10T17:41:24.419 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:41:24.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:41:24.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:41:24.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:41:24.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:41:24.952 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:41:25.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:41:25.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:41:25.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:41:25.355 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:41:25.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:41:25.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:41:25.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:41:25.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-10T17:41:25.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2024-09-10T17:41:26.527 INFO:tasks.workunit.client.0.smithi086.stderr:noout is unset 2024-09-10T17:41:26.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-10T17:41:26.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-86744-1-256 2024-09-10T17:41:26.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:41:26.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:41:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-86744-1-256 2024-09-10T17:41:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:41:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:41:26.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-86744-1-256 td/test-erasure-eio/COPY 2024-09-10T17:41:26.694 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:41:26.695 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:41:26.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:41:26.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:346: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:41:26.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:41:26.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:41:27.178 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:41:27.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:41:27.717 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:41:27.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:41:27.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:41:27.729 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:27.730 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:27.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:27.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:27.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:27.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:27.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:27.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:27.841 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:27.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:27.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:27.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:27.843 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:27.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:27.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:27.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:41:27.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:27.873 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:27.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:27.874 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:41:27.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:27.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:27.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:41:27.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:41:27.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:41:27.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:41:27.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:41:27.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:41:27.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:41:27.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:41:27.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:41:27.879 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:41:27.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:27.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:41:27.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:41:27.881 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:41:27.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:41:27.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:41:27.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:41:27.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:41:27.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:27.883 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:27.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:41:27.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:41:27.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:41:27.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:41:27.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:41:27.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:27.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:27.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:41:27.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:41:27.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:41:27.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:41:27.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:41:27.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:41:27.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:41:28.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:41:28.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:28.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:28.376 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:28.377 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:28.377 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:28.377 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:28.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:41:28.551 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:41:28.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:41:28.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:28.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:28.552 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:41:28.553 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:41:28.553 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:28.554 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:41:28.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:41:28.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:41:28.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:41:28.633 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:28.634 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:41:28.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:41:28.635 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:41:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:41:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:41:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:41:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:41:28.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:41:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:41:28.712 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:41:29.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:41:29.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:29.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:29.565 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:29.566 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:29.566 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:29.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:29.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:29.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:41:29.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:41:29.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:41:29.876 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:41:29.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:41:30.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:41:30.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:41:30.961 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:41:30.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:41:30.969 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:41:28.410+0000 7efce263bc80 0 load: jerasure load: lrc 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:276: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:41:30.970 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:41:30.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:41:30.976 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:41:30.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3d90de04-2536-4e39-8f82-199b10435200 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3d90de04-2536-4e39-8f82-199b10435200' 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 3d90de04-2536-4e39-8f82-199b10435200 2024-09-10T17:41:30.978 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:41:30.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKhOBmHw0pOxAA1DeUuxcLj1bDOdHfY9HWOw== 2024-09-10T17:41:30.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKhOBmHw0pOxAA1DeUuxcLj1bDOdHfY9HWOw=="}' 2024-09-10T17:41:30.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3d90de04-2536-4e39-8f82-199b10435200 -i td/test-erasure-eio/0/new.json 2024-09-10T17:41:31.261 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:41:31.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:41:31.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKhOBmHw0pOxAA1DeUuxcLj1bDOdHfY9HWOw== --osd-uuid 3d90de04-2536-4e39-8f82-199b10435200 2024-09-10T17:41:31.301 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:31.300+0000 7f30ed91b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:31.330 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:31.328+0000 7f30ed91b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:31.331 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:31.330+0000 7f30ed91b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:31.331 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:31.330+0000 7f30ed91b640 -1 bdev(0x56487d9dfc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:41:31.331 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:31.330+0000 7f30ed91b640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:41:38.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:41:38.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:41:38.002 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:41:38.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:41:38.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:41:38.537 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:41:38.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:41:38.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:41:38.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:41:38.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:41:38.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:41:38.565 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:38.564+0000 7fc27f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:38.596 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:38.592+0000 7fc27f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:38.596 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:38.593+0000 7fc27f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:38.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:41:38.977 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:41:38.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:38.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:39.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:39.937 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:39.936+0000 7fc27f017640 -1 Falling back to public interface 2024-09-10T17:41:40.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:40.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:40.401 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:41:40.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:41:40.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:40.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:40.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:41.511 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:41:41.510+0000 7fc27f017640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:41:41.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:41.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:41.835 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:41:41.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:41:41.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:41.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:42.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:43.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:43.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:43.237 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:41:43.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:41:43.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:43.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:43.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:44.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:44.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:44.629 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:41:44.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:41:44.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:44.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:45.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:46.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:46.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:46.034 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:41:46.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:41:46.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:46.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:46.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:47.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:47.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:47.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:41:47.632 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:41:47.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:47.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:48.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:49.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:49.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:49.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:41:49.042 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:41:49.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:49.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:49.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:50.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:50.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:50.610 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:41:50.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:41:50.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:50.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:51.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:52.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:52.032 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:52.032 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:41:52.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:41:52.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:52.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:52.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:53.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:53.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:53.426 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:41:53.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:41:53.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:53.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:53.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:54.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:54.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:54.820 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:41:54.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:41:54.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:54.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:55.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:56.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:56.516 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:41:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:41:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:56.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:57.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:41:58.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:41:58.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:41:58.324 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:41:58.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:41:58.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:41:58.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:41:59.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:00.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:00.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:00.209 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:42:00.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:42:00.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:00.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:00.926 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:01.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:01.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:01.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:42:01.927 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:42:01.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:01.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:02.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:03.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:03.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:03.405 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:42:03.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:42:03.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:03.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:03.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:04.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:04.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:04.972 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:42:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:42:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:04.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:42:05.450 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4154302811,v1:127.0.0.1:6803/4154302811] [v2:127.0.0.1:6804/4154302811,v1:127.0.0.1:6805/4154302811] exists,up 3d90de04-2536-4e39-8f82-199b10435200 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:42:05.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:05.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:42:05.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f1b145e-01b8-4e28-8046-55d5008a0e71 2024-09-10T17:42:05.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1f1b145e-01b8-4e28-8046-55d5008a0e71' 2024-09-10T17:42:05.458 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 1f1b145e-01b8-4e28-8046-55d5008a0e71 2024-09-10T17:42:05.458 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:42:05.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDthOBmLukmHBAAT2GTkCVmB0DdgmOekyYdow== 2024-09-10T17:42:05.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDthOBmLukmHBAAT2GTkCVmB0DdgmOekyYdow=="}' 2024-09-10T17:42:05.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f1b145e-01b8-4e28-8046-55d5008a0e71 -i td/test-erasure-eio/1/new.json 2024-09-10T17:42:06.039 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:42:06.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:42:06.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDthOBmLukmHBAAT2GTkCVmB0DdgmOekyYdow== --osd-uuid 1f1b145e-01b8-4e28-8046-55d5008a0e71 2024-09-10T17:42:06.080 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:06.079+0000 7f9d3de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:06.089 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:06.088+0000 7f9d3de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:06.090 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:06.089+0000 7f9d3de0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:06.091 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:06.090+0000 7f9d3de0d640 -1 bdev(0x557296520800 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:42:06.091 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:06.090+0000 7f9d3de0d640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:42:12.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:42:12.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:42:12.743 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:42:12.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:42:12.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:42:13.282 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:42:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:42:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:42:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:42:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:42:13.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:42:13.310 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:13.308+0000 7fc624d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:13.334 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:13.333+0000 7fc624d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:13.335 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:13.334+0000 7fc624d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:42:13.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:13.673 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:42:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:42:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:13.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:14.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:14.695 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:14.694+0000 7fc624d45640 -1 Falling back to public interface 2024-09-10T17:42:15.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:15.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:15.110 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:42:15.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:42:15.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:15.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:15.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:16.364 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:16.362+0000 7fc624d45640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:42:16.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:16.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:16.545 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:42:16.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:42:16.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:16.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:17.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:18.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:18.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:18.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:42:18.128 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:42:18.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:18.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:18.655 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:19.656 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:19.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:19.657 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:42:19.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:42:19.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:19.658 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:20.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:21.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:21.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:21.132 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:42:21.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:42:21.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:21.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:21.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:22.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:22.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:22.564 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:42:22.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:42:22.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:22.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:22.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:23.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:23.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:23.992 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:42:23.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:42:23.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:23.993 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:24.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:25.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:25.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:25.439 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:42:25.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:42:25.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:25.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:25.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:26.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:26.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:26.832 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:42:26.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:42:26.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:26.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:27.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:28.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:28.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:28.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:42:28.285 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:42:28.285 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:28.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:28.814 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:29.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:29.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:29.816 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:42:29.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:42:29.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:29.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:30.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:31.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:31.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:31.969 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:42:31.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:42:31.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:31.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:32.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:33.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:33.510 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:42:33.510 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:33.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:42:33.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:33.511 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:33.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:34.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:34.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:34.955 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:42:34.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:42:34.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:34.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:35.613 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:36.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:36.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:36.615 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:42:36.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:42:36.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:36.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:37.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:38.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:38.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:38.468 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:42:38.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:42:38.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:38.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:39.310 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:40.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:40.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:40.311 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:42:40.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:42:40.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:40.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:42:40.820 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3184479409,v1:127.0.0.1:6811/3184479409] [v2:127.0.0.1:6812/3184479409,v1:127.0.0.1:6813/3184479409] exists,up 1f1b145e-01b8-4e28-8046-55d5008a0e71 2024-09-10T17:42:40.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:42:40.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:42:40.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:42:40.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fb544a36-c811-4ad0-a3b3-4477a9a27f7c 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fb544a36-c811-4ad0-a3b3-4477a9a27f7c' 2024-09-10T17:42:40.828 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 fb544a36-c811-4ad0-a3b3-4477a9a27f7c 2024-09-10T17:42:40.829 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:42:40.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQheBm3FAwMhAA+0Qs62/iX0I9ABfpETCB/g== 2024-09-10T17:42:40.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQheBm3FAwMhAA+0Qs62/iX0I9ABfpETCB/g=="}' 2024-09-10T17:42:40.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fb544a36-c811-4ad0-a3b3-4477a9a27f7c -i td/test-erasure-eio/2/new.json 2024-09-10T17:42:41.287 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:42:41.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:42:41.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQheBm3FAwMhAA+0Qs62/iX0I9ABfpETCB/g== --osd-uuid fb544a36-c811-4ad0-a3b3-4477a9a27f7c 2024-09-10T17:42:41.329 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:41.327+0000 7fcaba019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:41.346 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:41.345+0000 7fcaba019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:41.347 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:41.346+0000 7fcaba019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:41.348 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:41.347+0000 7fcaba019640 -1 bdev(0x55d243f6fc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:42:41.348 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:41.347+0000 7fcaba019640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:42:47.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:42:47.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:42:48.000 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:42:48.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:42:48.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:42:48.543 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:42:48.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:42:48.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:42:48.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:42:48.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:42:48.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:42:48.570 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:48.569+0000 7fa29201c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:48.574 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:48.573+0000 7fa29201c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:48.575 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:48.574+0000 7fa29201c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:42:48.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:48.950 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:42:48.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:42:48.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:48.950 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:49.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:49.935 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:49.935+0000 7fa29201c640 -1 Falling back to public interface 2024-09-10T17:42:50.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:50.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:50.337 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:42:50.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:42:50.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:50.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:50.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:51.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:51.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:51.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:42:51.735 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:42:51.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:51.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:51.929 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:42:51.928+0000 7fa29201c640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:42:52.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:53.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:53.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:53.271 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:42:53.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:42:53.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:53.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:53.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:54.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:54.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:54.845 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:42:54.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:42:54.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:54.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:55.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:56.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:56.243 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:42:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:42:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:56.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:56.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:57.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:57.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:57.787 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:42:57.787 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:42:57.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:57.788 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:58.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:42:59.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:42:59.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:42:59.226 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:42:59.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:42:59.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:42:59.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:42:59.826 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:00.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:00.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:00.827 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:43:00.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:43:00.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:00.828 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:01.222 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:02.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:02.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:02.223 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:43:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:43:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:02.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:02.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:03.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:03.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:03.783 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:43:03.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:43:03.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:03.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:04.484 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:05.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:05.486 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:43:05.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:05.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:43:05.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:05.486 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:06.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:07.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:07.073 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:07.074 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:43:07.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:43:07.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:07.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:07.816 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:08.817 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:08.818 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:08.818 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:43:08.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:43:08.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:08.819 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:09.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:10.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:10.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:10.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:43:10.549 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:43:10.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:10.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:11.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:12.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:12.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:12.447 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:43:12.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:43:12.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:12.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:13.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:14.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:14.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:14.019 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:43:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:43:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:14.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:15.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:16.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:16.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:16.004 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:43:16.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:43:16.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:16.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:43:16.466 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2637994860,v1:127.0.0.1:6819/2637994860] [v2:127.0.0.1:6820/2637994860,v1:127.0.0.1:6821/2637994860] exists,up fb544a36-c811-4ad0-a3b3-4477a9a27f7c 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:43:16.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:43:16.471 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:43:16.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0fb0e97-12b7-4d18-812a-7fbed1836e30 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a0fb0e97-12b7-4d18-812a-7fbed1836e30' 2024-09-10T17:43:16.473 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 a0fb0e97-12b7-4d18-812a-7fbed1836e30 2024-09-10T17:43:16.474 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:43:16.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0heBm6ecIHRAAX3lRWeyWXf9g9rMTrd7YqQ== 2024-09-10T17:43:16.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0heBm6ecIHRAAX3lRWeyWXf9g9rMTrd7YqQ=="}' 2024-09-10T17:43:16.488 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0fb0e97-12b7-4d18-812a-7fbed1836e30 -i td/test-erasure-eio/3/new.json 2024-09-10T17:43:17.303 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:43:17.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:43:17.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0heBm6ecIHRAAX3lRWeyWXf9g9rMTrd7YqQ== --osd-uuid a0fb0e97-12b7-4d18-812a-7fbed1836e30 2024-09-10T17:43:17.345 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:17.344+0000 7fe4e3754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:17.347 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:17.346+0000 7fe4e3754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:17.348 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:17.347+0000 7fe4e3754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:17.348+0000 7fe4e3754640 -1 bdev(0x55c2cb35fc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:43:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:17.348+0000 7fe4e3754640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:43:23.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:43:23.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:43:23.792 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:43:23.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:43:23.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:43:24.327 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:43:24.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:43:24.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:43:24.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:43:24.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:43:24.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:43:24.354 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:24.353+0000 7f75a0814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:24.375 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:24.374+0000 7f75a0814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:24.376 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:24.375+0000 7f75a0814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:43:24.746 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:24.747 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:43:24.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:43:24.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:24.748 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:25.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:26.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:26.166 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:26.166 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:43:26.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:43:26.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:26.167 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:26.243 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:26.242+0000 7f75a0814640 -1 Falling back to public interface 2024-09-10T17:43:26.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:27.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:27.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:27.678 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:43:27.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:43:27.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:27.679 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:27.945 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:43:27.944+0000 7f75a0814640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:43:28.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:29.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:29.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:29.038 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:43:29.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:43:29.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:29.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:29.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:30.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:30.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:30.704 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:43:30.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:43:30.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:30.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:31.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:32.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:32.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:32.243 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:43:32.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:43:32.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:32.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:32.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:33.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:33.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:33.636 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:43:33.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:43:33.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:33.637 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:34.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:35.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:35.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:35.293 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:43:35.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:43:35.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:35.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:35.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:36.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:36.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:36.691 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:43:36.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:43:36.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:36.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:37.223 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:38.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:38.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:38.225 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:43:38.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:43:38.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:38.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:38.618 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:39.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:39.619 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:39.619 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:43:39.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:43:39.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:39.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:40.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:41.092 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:41.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:41.093 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:43:41.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:43:41.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:41.094 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:41.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:42.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:42.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:42.548 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:43:42.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:43:42.548 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:42.549 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:43.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:44.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:44.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:44.265 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:43:44.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:43:44.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:44.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:45.193 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:46.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:46.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:46.195 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:43:46.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:43:46.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:46.196 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:47.491 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:48.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:48.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:48.493 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:43:48.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:43:48.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:48.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:48.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:49.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:49.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:49.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:43:49.887 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:43:49.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:49.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:51.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:52.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:52.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:52.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:43:52.108 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:43:52.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:52.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:52.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:43:53.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:43:53.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:43:53.616 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:43:53.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:43:53.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:43:53.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:43:54.174 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3791083477,v1:127.0.0.1:6827/3791083477] [v2:127.0.0.1:6828/3791083477,v1:127.0.0.1:6829/3791083477] exists,up a0fb0e97-12b7-4d18-812a-7fbed1836e30 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:43:54.177 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:43:54.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:43:54.256 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:43:54.263 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:43:54.266 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:41:39.940+0000 7fc27f017640 0 load: jerasure load: lrc 2024-09-10T17:43:54.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:278: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-10T17:43:54.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:279: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:43:54.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:43:54.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:43:54.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:43:54.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:43:54.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:43:54.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:43:54.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:43:55.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:43:55.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:43:55.914 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:43:55.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:43:56.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:43:56.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:43:56.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:43:56.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:43:56.927 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:43:56.928 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:43:56.928 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:43:56.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:43:56.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:43:56.928 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:43:57.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:43:57.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:43:57.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:43:57.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:43:57.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:43:57.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:43:57.547 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:43:57.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803807 2024-09-10T17:43:57.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803807 2024-09-10T17:43:57.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807' 2024-09-10T17:43:57.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:43:57.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:43:57.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574877 2024-09-10T17:43:57.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574877 2024-09-10T17:43:57.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574877' 2024-09-10T17:43:57.803 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:43:57.803 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:43:57.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345935 2024-09-10T17:43:57.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345935 2024-09-10T17:43:57.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574877 2-85899345935' 2024-09-10T17:43:57.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:43:57.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018884 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018884 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574877 2-85899345935 3-128849018884' 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803807 2024-09-10T17:43:58.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:43:58.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:43:58.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803807 2024-09-10T17:43:58.029 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:43:58.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803807 2024-09-10T17:43:58.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-10T17:43:58.030 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803807 2024-09-10T17:43:58.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:43:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803807 2024-09-10T17:43:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:43:59.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:43:59.444 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:43:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803807 -lt 25769803807 2024-09-10T17:43:59.836 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:43:59.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574877 2024-09-10T17:43:59.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:43:59.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:43:59.837 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574877 2024-09-10T17:43:59.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:43:59.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574877 2024-09-10T17:43:59.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-10T17:43:59.839 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574877 2024-09-10T17:43:59.839 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:44:00.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-10T17:44:00.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:44:00.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-85899345935 2024-09-10T17:44:00.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:44:00.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:44:00.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-85899345935 2024-09-10T17:44:00.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:44:00.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345935 2024-09-10T17:44:00.290 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 85899345935 2024-09-10T17:44:00.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-09-10T17:44:00.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:44:00.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345935 -lt 85899345935 2024-09-10T17:44:00.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:44:00.705 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-128849018884 2024-09-10T17:44:00.708 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-128849018884 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018884 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 128849018884' 2024-09-10T17:44:00.710 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 128849018884 2024-09-10T17:44:00.711 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:44:01.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018885 -lt 128849018884 2024-09-10T17:44:01.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:44:01.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:44:01.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:44:01.660 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:44:01.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:44:01.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:44:02.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:44:02.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:44:02.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:44:02.034 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-10T17:44:02.556 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-86744-0 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-86744-0 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-86744-0 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:44:02.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:44:02.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:44:02.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:44:02.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:44:02.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:44:02.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-86744-0 td/test-erasure-eio/ORIGINAL 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-0 td/test-erasure-eio 0 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-0 2024-09-10T17:44:02.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-0 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-0 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-0 2024-09-10T17:44:02.710 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:44:03.137 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:44:03.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:44:03.141 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:44:03.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:44:03.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:44:03.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:44:03.226 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:44:03.226 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:44:03.226 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:44:03.226 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:03.227 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:03.227 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:03.227 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:44:03.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:44:03.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-86744-0 0 2024-09-10T17:44:03.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-0 2024-09-10T17:44:03.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:44:03.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:44:03.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-0 2024-09-10T17:44:03.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:44:03.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:44:03.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-86744-0 td/test-erasure-eio/COPY 2024-09-10T17:44:03.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:44:03.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:44:03.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-10T17:44:03.344 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-10T17:44:03.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=1 2024-09-10T17:44:03.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-0 td/test-erasure-eio 1 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-0 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:44:03.347 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-0 2024-09-10T17:44:03.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:44:03.348 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-0 2024-09-10T17:44:03.348 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-0 2024-09-10T17:44:03.348 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:44:03.789 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:44:03.789 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:44:03.789 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:44:03.790 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-10T17:44:03.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:44:03.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T17:44:03.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:44:03.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T17:44:03.791 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:44:03.792 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:44:03.792 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:44:03.792 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T17:44:03.792 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:44:03.792 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:44:03.793 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:03.793 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:03.793 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:03.793 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:44:03.793 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:44:03.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:44:03.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:44:03.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:44:03.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-86744-0 1 2024-09-10T17:44:03.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 0 -eq 1 -a 1 -eq 2 ']' 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-0 fail 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-0 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-10T17:44:03.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-86744-0 td/test-erasure-eio/COPY 2024-09-10T17:44:04.023 INFO:tasks.workunit.client.0.smithi086.stderr:error getting pool-jerasure/obj-eio-86744-0: (5) Input/output error 2024-09-10T17:44:04.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-10T17:44:04.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:44:04.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:44:04.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:44:04.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:44:04.501 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:44:04.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:44:05.077 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:44:05.089 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:44:05.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:44:05.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:44:05.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:44:05.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:44:05.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:44:05.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:44:05.199 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:44:05.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:44:05.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:44:05.201 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:44:05.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:05.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:44:05.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:44:05.203 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:05.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:44:05.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:44:05.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:44:05.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:44:05.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.224 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.224 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:44:05.227 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:44:05.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:44:05.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:44:05.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:44:05.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:44:05.229 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:44:05.229 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:44:05.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:44:05.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:44:05.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:44:05.230 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:44:05.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:05.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:44:05.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:44:05.234 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:44:05.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:44:05.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:44:05.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:44:05.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:44:05.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.237 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.238 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:44:05.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.697 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:05.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:44:05.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:44:05.884 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:44:05.884 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:44:05.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:44:05.961 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:44:05.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:44:05.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:44:05.962 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:44:05.963 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:44:05.964 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:44:06.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:44:06.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:44:06.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:44:06.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:44:06.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:44:06.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:44:06.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:44:06.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:06.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:44:06.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:44:06.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:44:06.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:44:07.197 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:44:07.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:44:08.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:44:08.282 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:44:08.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:44:08.290 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:44:05.728+0000 7f12c060ec80 0 load: jerasure load: lrc 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:44:08.291 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:44:08.292 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:44:08.297 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:08.298 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:44:08.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1ff9e97b-b605-4e66-bb81-a7a03caca654 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1ff9e97b-b605-4e66-bb81-a7a03caca654' 2024-09-10T17:44:08.300 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 1ff9e97b-b605-4e66-bb81-a7a03caca654 2024-09-10T17:44:08.301 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:44:08.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoheBm1UW5EhAAPgzHgQ4yMN4YyJxv2tio5w== 2024-09-10T17:44:08.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoheBm1UW5EhAAPgzHgQ4yMN4YyJxv2tio5w=="}' 2024-09-10T17:44:08.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1ff9e97b-b605-4e66-bb81-a7a03caca654 -i td/test-erasure-eio/0/new.json 2024-09-10T17:44:08.574 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:44:08.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:44:08.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoheBm1UW5EhAAPgzHgQ4yMN4YyJxv2tio5w== --osd-uuid 1ff9e97b-b605-4e66-bb81-a7a03caca654 2024-09-10T17:44:08.614 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:08.612+0000 7fee9fb43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:08.667 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:08.666+0000 7fee9fb43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:08.668 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:08.667+0000 7fee9fb43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:08.669 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:08.668+0000 7fee9fb43640 -1 bdev(0x55f13bc4dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:44:08.669 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:08.668+0000 7fee9fb43640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:44:14.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:44:14.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:44:14.572 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:44:14.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:44:14.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:44:15.105 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:44:15.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:44:15.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:44:15.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:44:15.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:44:15.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:44:15.133 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:15.131+0000 7efe8dab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:15.134 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:15.133+0000 7efe8dab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:15.135 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:15.134+0000 7efe8dab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:15.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:44:15.527 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:44:15.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:15.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:15.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:16.227 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:16.226+0000 7efe8dab6640 -1 Falling back to public interface 2024-09-10T17:44:16.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:16.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:16.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:44:16.954 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:44:16.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:16.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:17.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:17.940 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:17.938+0000 7efe8dab6640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:44:18.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:18.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:18.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:44:18.388 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:44:18.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:18.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:18.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:19.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:19.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:19.918 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:44:19.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:44:19.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:19.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:20.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:21.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:21.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:21.313 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:44:21.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:44:21.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:21.313 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:21.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:22.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:22.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:22.764 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:44:22.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:44:22.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:22.764 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:23.182 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:24.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:24.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:24.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:44:24.185 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:44:24.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:24.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:24.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:25.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:25.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:25.718 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:44:25.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:44:25.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:25.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:26.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:27.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:27.116 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:44:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:44:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:27.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:27.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:28.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:28.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:28.509 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:44:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:44:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:28.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:28.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:29.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:29.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:29.911 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:44:29.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:44:29.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:29.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:30.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:31.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:31.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:31.301 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:44:31.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:44:31.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:31.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:32.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:33.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:33.123 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:44:33.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:33.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:44:33.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:33.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:34.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:35.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:35.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:35.350 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:44:35.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:44:35.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:35.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:36.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:37.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:37.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:37.209 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:44:37.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:44:37.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:37.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:38.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:39.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:39.038 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:44:39.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:39.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:44:39.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:39.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:39.843 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:40.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:40.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:40.845 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:44:40.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:44:40.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:40.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:44:41.282 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2539387231,v1:127.0.0.1:6803/2539387231] [v2:127.0.0.1:6804/2539387231,v1:127.0.0.1:6805/2539387231] exists,up 1ff9e97b-b605-4e66-bb81-a7a03caca654 2024-09-10T17:44:41.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:44:41.286 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:44:41.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:41.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:44:41.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:44:41.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a429e690-6b10-489c-9486-86e405e1a9b8 2024-09-10T17:44:41.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a429e690-6b10-489c-9486-86e405e1a9b8' 2024-09-10T17:44:41.290 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 a429e690-6b10-489c-9486-86e405e1a9b8 2024-09-10T17:44:41.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:44:41.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCJheBmpBkdEhAAI8kgF2YcmvtsFSBTdQl9VQ== 2024-09-10T17:44:41.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCJheBmpBkdEhAAI8kgF2YcmvtsFSBTdQl9VQ=="}' 2024-09-10T17:44:41.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a429e690-6b10-489c-9486-86e405e1a9b8 -i td/test-erasure-eio/1/new.json 2024-09-10T17:44:41.917 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:44:41.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:44:41.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJheBmpBkdEhAAI8kgF2YcmvtsFSBTdQl9VQ== --osd-uuid a429e690-6b10-489c-9486-86e405e1a9b8 2024-09-10T17:44:41.959 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:41.958+0000 7fbe1614d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:41.966 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:41.965+0000 7fbe1614d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:41.967 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:41.966+0000 7fbe1614d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:41.968 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:41.967+0000 7fbe1614d640 -1 bdev(0x559a95311c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:44:41.968 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:41.967+0000 7fbe1614d640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:44:48.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:44:48.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:44:48.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:44:48.265 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:44:48.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:44:48.808 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:44:48.808 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:44:48.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:44:48.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:44:48.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:44:48.809 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:44:48.835 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:48.834+0000 7f9557e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:48.848 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:48.846+0000 7f9557e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:48.848 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:48.847+0000 7f9557e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:44:49.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:44:49.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:44:49.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:44:49.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:44:49.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:44:49.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:49.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:44:49.272 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:44:49.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:49.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:49.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:50.439 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:50.438+0000 7f9557e0e640 -1 Falling back to public interface 2024-09-10T17:44:50.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:50.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:50.706 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:44:50.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:44:50.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:50.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:51.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:52.042 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:44:52.041+0000 7f9557e0e640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:44:52.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:52.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:52.099 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:44:52.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:44:52.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:52.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:52.687 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:53.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:53.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:53.689 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:44:53.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:44:53.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:53.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:54.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:55.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:55.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:55.176 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:44:55.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:44:55.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:55.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:55.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:56.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:56.599 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:56.600 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:44:56.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:44:56.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:56.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:56.994 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:57.995 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:57.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:57.996 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:44:57.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:44:57.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:57.996 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:58.418 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:44:59.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:44:59.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:44:59.420 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:44:59.420 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:44:59.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:44:59.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:44:59.927 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:00.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:00.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:00.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:45:00.929 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:45:00.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:00.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:01.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:02.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:02.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:02.351 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:45:02.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:45:02.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:02.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:02.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:03.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:03.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:03.824 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:45:03.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:45:03.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:03.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:04.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:05.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:05.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:05.304 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:45:05.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:45:05.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:05.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:06.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:07.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:07.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:07.417 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:45:07.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:45:07.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:07.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:08.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:09.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:09.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:09.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:45:09.296 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:45:09.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:09.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:09.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:10.987 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:10.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:10.988 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:45:10.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:45:10.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:10.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:11.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:12.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:12.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:12.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:45:12.892 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:45:12.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:12.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:13.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:14.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:14.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:14.582 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:45:14.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:45:14.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:14.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:15.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:16.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:16.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:16.048 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:45:16.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:45:16.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:16.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:45:16.497 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2121719195,v1:127.0.0.1:6811/2121719195] [v2:127.0.0.1:6812/2121719195,v1:127.0.0.1:6813/2121719195] exists,up a429e690-6b10-489c-9486-86e405e1a9b8 2024-09-10T17:45:16.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:45:16.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:45:16.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:45:16.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:45:16.503 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:45:16.504 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:45:16.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a7eeda56-4530-4664-a4f1-04a91379aac1 2024-09-10T17:45:16.505 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 a7eeda56-4530-4664-a4f1-04a91379aac1 2024-09-10T17:45:16.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a7eeda56-4530-4664-a4f1-04a91379aac1' 2024-09-10T17:45:16.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:45:16.519 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCsheBmPzjnHhAA8mJjDwI32+k3tryLMDic/Q== 2024-09-10T17:45:16.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCsheBmPzjnHhAA8mJjDwI32+k3tryLMDic/Q=="}' 2024-09-10T17:45:16.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a7eeda56-4530-4664-a4f1-04a91379aac1 -i td/test-erasure-eio/2/new.json 2024-09-10T17:45:17.004 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:45:17.015 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:45:17.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsheBmPzjnHhAA8mJjDwI32+k3tryLMDic/Q== --osd-uuid a7eeda56-4530-4664-a4f1-04a91379aac1 2024-09-10T17:45:17.046 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:17.045+0000 7fa25bedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:17.089 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:17.088+0000 7fa25bedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:17.090 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:17.089+0000 7fa25bedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:17.091 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:17.090+0000 7fa25bedc640 -1 bdev(0x564050d5fc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:45:17.091 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:17.090+0000 7fa25bedc640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:45:23.324 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:45:23.325 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:45:23.325 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:45:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:45:23.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:45:23.861 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:45:23.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:45:23.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:45:23.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:45:23.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:45:23.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:45:23.889 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:23.888+0000 7fcbc5b54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:23.908 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:23.907+0000 7fcbc5b54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:23.909 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:23.908+0000 7fcbc5b54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:24.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:45:24.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:45:24.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:45:24.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:45:24.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:45:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:45:24.332 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:45:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:24.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:24.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:25.266 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:25.265+0000 7fcbc5b54640 -1 Falling back to public interface 2024-09-10T17:45:25.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:25.749 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:25.750 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:45:25.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:45:25.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:25.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:26.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:26.887 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:26.885+0000 7fcbc5b54640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:45:27.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:27.185 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:27.185 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:45:27.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:45:27.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:27.186 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:27.673 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:28.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:28.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:28.675 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:45:28.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:45:28.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:28.676 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:29.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:30.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:30.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:30.162 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:45:30.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:45:30.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:30.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:30.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:31.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:31.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:31.608 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:45:31.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:45:31.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:31.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:32.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:33.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:33.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:33.007 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:45:33.007 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:45:33.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:33.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:33.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:34.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:34.433 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:45:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:45:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:34.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:34.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:35.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:35.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:35.854 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:45:35.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:45:35.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:35.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:36.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:37.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:37.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:37.269 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:45:37.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:45:37.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:37.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:37.750 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:38.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:38.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:38.752 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:45:38.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:45:38.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:38.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:39.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:40.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:40.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:40.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:45:40.219 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:45:40.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:40.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:40.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:41.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:41.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:41.916 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:45:41.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:45:41.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:41.917 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:42.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:43.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:43.844 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:43.844 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:45:43.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:45:43.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:43.845 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:44.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:45.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:45.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:45.588 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:45:45.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:45:45.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:45.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:46.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:47.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:47.456 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:47.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:45:47.457 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:45:47.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:47.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:48.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:49.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:49.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:49.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:45:49.022 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:45:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:49.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:49.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:45:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:45:50.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:50.918 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:45:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:45:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:50.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:45:51.510 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3271662219,v1:127.0.0.1:6819/3271662219] [v2:127.0.0.1:6820/3271662219,v1:127.0.0.1:6821/3271662219] exists,up a7eeda56-4530-4664-a4f1-04a91379aac1 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:45:51.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:45:51.516 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:45:51.517 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:45:51.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:45:51.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2191e786-ef94-46de-bf5b-0fd15742beb2 2024-09-10T17:45:51.518 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2191e786-ef94-46de-bf5b-0fd15742beb2' 2024-09-10T17:45:51.518 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 2191e786-ef94-46de-bf5b-0fd15742beb2 2024-09-10T17:45:51.518 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:45:51.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPheBmuF+4HxAAwt3BQgi1Ah0hR0no35XY6g== 2024-09-10T17:45:51.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPheBmuF+4HxAAwt3BQgi1Ah0hR0no35XY6g=="}' 2024-09-10T17:45:51.533 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2191e786-ef94-46de-bf5b-0fd15742beb2 -i td/test-erasure-eio/3/new.json 2024-09-10T17:45:52.742 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:45:52.753 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:45:52.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPheBmuF+4HxAAwt3BQgi1Ah0hR0no35XY6g== --osd-uuid 2191e786-ef94-46de-bf5b-0fd15742beb2 2024-09-10T17:45:52.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:52.783+0000 7fb458426640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:52.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:52.785+0000 7fb458426640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:52.787 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:52.786+0000 7fb458426640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:52.787 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:52.786+0000 7fb458426640 -1 bdev(0x55b6bbf27c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:45:52.787 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:52.786+0000 7fb458426640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:45:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:45:58.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:45:58.744 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:45:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:45:58.745 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:45:59.281 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:45:59.281 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:45:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:45:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:45:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:45:59.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:45:59.308 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:59.307+0000 7f9d805fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:59.310 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:59.309+0000 7f9d805fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:59.311 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:45:59.310+0000 7f9d805fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:45:59.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:45:59.707 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:45:59.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:45:59.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:00.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:01.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:01.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:01.133 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:46:01.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:46:01.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:01.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:01.159 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:01.158+0000 7f9d805fd640 -1 Falling back to public interface 2024-09-10T17:46:01.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:02.479 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:02.477+0000 7f9d805fd640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:46:02.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:02.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:02.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:46:02.561 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:46:02.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:02.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:03.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:04.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:04.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:04.338 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:46:04.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:46:04.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:04.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:04.894 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:05.895 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:05.896 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:46:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:46:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:05.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:06.384 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:07.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:07.385 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:07.386 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:46:07.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:46:07.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:07.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:07.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:08.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:08.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:08.796 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:46:08.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:46:08.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:08.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:09.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:10.183 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:10.183 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:46:10.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:46:10.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:10.184 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:10.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:11.702 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:11.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:11.703 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:46:11.703 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:46:11.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:11.704 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:12.116 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:13.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:13.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:13.117 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:46:13.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:46:13.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:13.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:13.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:14.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:14.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:14.540 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:46:14.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:46:14.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:14.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:14.953 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:15.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:15.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:15.955 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:46:15.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:46:15.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:15.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:16.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:46:17.349 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:46:17.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:17.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:17.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:18.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:18.983 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:18.983 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:46:18.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:46:18.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:18.984 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:19.986 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:20.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:20.988 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:20.988 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:46:20.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:46:20.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:20.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:21.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:22.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:22.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:22.795 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:46:22.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:46:22.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:22.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:23.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:24.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:24.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:24.421 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:46:24.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:46:24.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:24.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:25.230 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:26.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:26.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:26.232 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:46:26.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:46:26.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:26.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:27.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:28.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:28.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:28.045 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:46:28.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:46:28.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:28.046 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:46:28.559 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1808563430,v1:127.0.0.1:6827/1808563430] [v2:127.0.0.1:6828/1808563430,v1:127.0.0.1:6829/1808563430] exists,up 2191e786-ef94-46de-bf5b-0fd15742beb2 2024-09-10T17:46:28.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:46:28.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:46:28.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:46:28.560 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:46:28.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:46:28.639 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:46:28.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:46:28.649 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:44:16.230+0000 7efe8dab6640 0 load: jerasure load: lrc 2024-09-10T17:46:28.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-10T17:46:28.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:291: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:46:28.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:46:28.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:46:29.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:46:29.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:46:30.708 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:46:30.719 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:46:31.721 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:46:31.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:46:31.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:46:31.722 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:46:31.722 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:46:31.723 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:46:31.723 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:46:31.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:46:31.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:46:31.723 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:46:31.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:46:32.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:46:32.241 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:46:32.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803805 2024-09-10T17:46:32.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803805 2024-09-10T17:46:32.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805' 2024-09-10T17:46:32.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:46:32.356 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:46:32.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=51539607577 2024-09-10T17:46:32.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 51539607577 2024-09-10T17:46:32.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-51539607577' 2024-09-10T17:46:32.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:46:32.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:46:32.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411345 2024-09-10T17:46:32.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411345 2024-09-10T17:46:32.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-51539607577 2-77309411345' 2024-09-10T17:46:32.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:46:32.579 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084298 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084298 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-51539607577 2-77309411345 3-120259084298' 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803805 2024-09-10T17:46:32.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:46:32.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:46:32.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803805 2024-09-10T17:46:32.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:46:32.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803805 2024-09-10T17:46:32.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-10T17:46:32.693 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803805 2024-09-10T17:46:32.694 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:46:33.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-10T17:46:33.082 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:46:33.082 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-51539607577 2024-09-10T17:46:33.083 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:46:33.083 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:46:33.084 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-51539607577 2024-09-10T17:46:33.084 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:46:33.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=51539607577 2024-09-10T17:46:33.085 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 51539607577' 2024-09-10T17:46:33.085 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 51539607577 2024-09-10T17:46:33.085 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:46:33.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 51539607577 -lt 51539607577 2024-09-10T17:46:33.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:46:33.467 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411345 2024-09-10T17:46:33.467 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:46:33.467 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:46:33.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411345 2024-09-10T17:46:33.468 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:46:33.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411345 2024-09-10T17:46:33.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411345' 2024-09-10T17:46:33.470 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411345 2024-09-10T17:46:33.470 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:46:33.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411345 -lt 77309411345 2024-09-10T17:46:33.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:46:33.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084298 2024-09-10T17:46:33.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:46:33.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:46:33.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084298 2024-09-10T17:46:33.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:46:33.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084298 2024-09-10T17:46:33.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-09-10T17:46:33.890 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084298 2024-09-10T17:46:33.890 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:46:34.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084298 -lt 120259084298 2024-09-10T17:46:34.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:46:34.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:46:34.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:46:34.825 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:46:34.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:46:34.826 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:46:35.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:46:35.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:46:35.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:46:35.200 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:293: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:294: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-10T17:46:35.724 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-86744-1 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-86744-1 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-86744-1 2024-09-10T17:46:35.725 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:46:35.726 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-86744-1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:46:35.899 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-1 td/test-erasure-eio 1 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-1 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:46:35.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-1 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-1 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-1 2024-09-10T17:46:35.901 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:46:36.286 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:46:36.286 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:46:36.287 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T17:46:36.289 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:46:36.290 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:46:36.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:46:36.374 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:46:36.375 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:46:36.376 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-86744-1 1 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-1 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-1 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:46:36.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-86744-1 td/test-erasure-eio/COPY 2024-09-10T17:46:36.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:46:36.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:46:36.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-10T17:46:36.494 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-10T17:46:36.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-1 td/test-erasure-eio 2 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-1 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:46:36.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=2 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-1 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-1 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-1 2024-09-10T17:46:36.498 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:46:36.912 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=2 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:46:36.913 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/2/type 2024-09-10T17:46:36.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:46:36.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=2 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.2 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T17:46:36.915 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T17:46:36.916 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:36.916 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:36.916 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:36.916 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.2.asok 2024-09-10T17:46:36.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:46:37.000 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:46:37.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.2 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:37.002 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:37.003 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.2.asok 2024-09-10T17:46:37.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:46:37.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-86744-1 2 2024-09-10T17:46:37.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-10T17:46:37.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-1 2024-09-10T17:46:37.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:46:37.089 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:46:37.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-1 2024-09-10T17:46:37.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:46:37.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:46:37.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-86744-1 td/test-erasure-eio/COPY 2024-09-10T17:46:37.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:46:37.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:46:37.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:46:37.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:295: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:46:37.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:46:37.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:46:37.600 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:46:37.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:46:38.237 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:46:38.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:46:38.358 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:46:38.358 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:46:38.359 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:46:38.359 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:46:38.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:46:38.361 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:46:38.361 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:46:38.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:38.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:46:38.362 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:46:38.362 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:38.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:46:38.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:46:38.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:46:38.383 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:46:38.383 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:38.383 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:38.383 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:46:38.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:46:38.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:46:38.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:46:38.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:46:38.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:46:38.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:46:38.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:46:38.389 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:46:38.389 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:46:38.390 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:46:38.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:46:38.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:46:38.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:46:38.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:46:38.393 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:38.394 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:38.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:46:38.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:46:38.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:46:38.394 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:46:38.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:46:38.395 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:38.396 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:38.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:46:38.398 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:46:38.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:46:38.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:46:38.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:46:38.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:46:38.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:46:38.845 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:46:38.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:46:39.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:46:39.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:46:39.022 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:46:39.022 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:46:39.022 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:39.023 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:39.024 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:39.024 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:46:39.024 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:46:39.024 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:46:39.100 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:39.101 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:46:39.102 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:46:39.102 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:46:39.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:46:40.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:46:40.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:46:40.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:46:40.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:46:40.056 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:46:40.354 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:46:40.363 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:46:41.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:46:41.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:46:41.438 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:46:41.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:46:41.446 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:46:38.876+0000 7faad79c5c80 0 load: jerasure load: lrc 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_subread_missing: setup_osds 4 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:46:41.447 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:46:41.448 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:46:41.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:46:41.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:46:41.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:46:41.454 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=67183ef3-5beb-4290-bcb3-eca047af385e 2024-09-10T17:46:41.455 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 67183ef3-5beb-4290-bcb3-eca047af385e 2024-09-10T17:46:41.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 67183ef3-5beb-4290-bcb3-eca047af385e' 2024-09-10T17:46:41.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:46:41.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQABhuBmyWwIHBAAZvux2Eo/GbHVKMd2JgtlGg== 2024-09-10T17:46:41.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQABhuBmyWwIHBAAZvux2Eo/GbHVKMd2JgtlGg=="}' 2024-09-10T17:46:41.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 67183ef3-5beb-4290-bcb3-eca047af385e -i td/test-erasure-eio/0/new.json 2024-09-10T17:46:41.756 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:46:41.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:46:41.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABhuBmyWwIHBAAZvux2Eo/GbHVKMd2JgtlGg== --osd-uuid 67183ef3-5beb-4290-bcb3-eca047af385e 2024-09-10T17:46:41.795 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:41.794+0000 7f1aaf45e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:41.824 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:41.823+0000 7f1aaf45e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:41.825 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:41.824+0000 7f1aaf45e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:41.826 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:41.825+0000 7f1aaf45e640 -1 bdev(0x55e6cda5fc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:46:41.826 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:41.825+0000 7f1aaf45e640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:46:47.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:46:47.779 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:46:47.780 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:46:47.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:46:47.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:46:48.310 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:46:48.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:46:48.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:46:48.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:46:48.311 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:46:48.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:46:48.338 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:48.337+0000 7f1949fb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:48.362 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:48.361+0000 7f1949fb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:48.363 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:48.362+0000 7f1949fb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:46:48.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:46:48.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:46:48.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:46:48.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:48.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:49.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:49.976 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:49.975+0000 7f1949fb0640 -1 Falling back to public interface 2024-09-10T17:46:50.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:50.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:50.178 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:46:50.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:46:50.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:50.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:50.602 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:51.547 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:46:51.546+0000 7f1949fb0640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:46:51.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:51.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:51.604 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:46:51.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:46:51.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:51.604 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:52.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:53.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:53.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:53.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:46:53.160 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:46:53.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:53.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:53.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:54.589 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:54.590 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:46:54.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:54.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:46:54.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:54.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:54.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:55.998 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:55.998 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:46:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:46:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:55.999 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:56.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:57.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:57.412 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:46:57.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:57.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:46:57.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:57.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:57.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:46:58.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:46:58.823 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:46:58.823 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:46:58.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:46:58.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:46:58.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:46:59.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:00.253 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:00.254 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:47:00.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:00.254 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:47:00.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:00.255 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:00.682 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:01.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:01.683 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:47:01.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:01.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:47:01.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:01.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:02.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:03.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:03.107 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:47:03.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:03.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:47:03.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:03.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:03.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:04.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:04.716 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:47:04.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:04.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:47:04.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:04.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:05.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:06.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:06.629 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:47:06.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:06.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:47:06.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:06.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:07.538 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:08.539 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:08.540 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:47:08.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:08.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:47:08.540 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:08.541 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:10.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:10.217 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:47:10.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:10.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:47:10.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:10.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:11.001 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:12.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:12.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:12.004 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:47:12.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:47:12.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:12.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:12.643 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:13.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:13.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:13.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:47:13.646 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:47:13.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:13.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:47:14.142 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2278175877,v1:127.0.0.1:6803/2278175877] [v2:127.0.0.1:6804/2278175877,v1:127.0.0.1:6805/2278175877] exists,up 67183ef3-5beb-4290-bcb3-eca047af385e 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:47:14.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:47:14.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:47:14.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ae69166-b405-4e89-aef6-1a5dce2c00ba 2024-09-10T17:47:14.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7ae69166-b405-4e89-aef6-1a5dce2c00ba' 2024-09-10T17:47:14.150 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 7ae69166-b405-4e89-aef6-1a5dce2c00ba 2024-09-10T17:47:14.150 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:47:14.164 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAihuBmhv+/CRAAvDITaPCMLZuU4f3qPjGPkw== 2024-09-10T17:47:14.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAihuBmhv+/CRAAvDITaPCMLZuU4f3qPjGPkw=="}' 2024-09-10T17:47:14.165 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ae69166-b405-4e89-aef6-1a5dce2c00ba -i td/test-erasure-eio/1/new.json 2024-09-10T17:47:14.657 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:47:14.668 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:47:14.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAihuBmhv+/CRAAvDITaPCMLZuU4f3qPjGPkw== --osd-uuid 7ae69166-b405-4e89-aef6-1a5dce2c00ba 2024-09-10T17:47:14.698 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:14.697+0000 7fc309818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:14.703 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:14.701+0000 7fc309818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:14.703 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:14.702+0000 7fc309818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:14.703 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:14.702+0000 7fc309818640 -1 bdev(0x55be2ae69c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:47:14.703 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:14.702+0000 7fc309818640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:47:21.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:47:21.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:47:21.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:47:21.055 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:47:21.055 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:47:21.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:47:21.587 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:47:21.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:47:21.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:47:21.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:47:21.588 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:47:21.614 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:21.613+0000 7f337d01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:21.638 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:21.638+0000 7f337d01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:21.639 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:21.639+0000 7f337d01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:22.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:47:22.009 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:47:22.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:22.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:22.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:22.982 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:22.981+0000 7f337d01b640 -1 Falling back to public interface 2024-09-10T17:47:23.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:23.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:23.429 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:47:23.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:47:23.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:23.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:23.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:24.692 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:24.691+0000 7f337d01b640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:47:24.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:24.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:24.867 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:47:24.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:47:24.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:24.868 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:25.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:26.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:26.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:26.346 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:47:26.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:47:26.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:26.347 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:26.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:27.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:27.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:27.841 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:47:27.842 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:47:27.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:27.842 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:28.239 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:29.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:29.240 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:29.241 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:47:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:47:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:29.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:30.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:30.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:30.780 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:47:30.780 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:47:30.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:30.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:31.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:32.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:32.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:32.178 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:47:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:47:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:32.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:32.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:33.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:33.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:33.571 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:47:33.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:47:33.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:33.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:33.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:34.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:34.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:34.967 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:47:34.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:47:34.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:34.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:35.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:36.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:36.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:36.480 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:47:36.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:47:36.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:36.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:37.031 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:38.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:38.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:38.033 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:47:38.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:47:38.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:38.034 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:38.608 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:39.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:39.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:39.610 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:47:39.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:47:39.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:39.611 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:40.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:41.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:41.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:41.381 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:47:41.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:47:41.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:41.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:42.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:43.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:43.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:43.071 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:47:43.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:47:43.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:43.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:43.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:44.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:44.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:44.969 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:47:44.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:47:44.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:44.970 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:45.754 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:46.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:46.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:46.757 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:47:46.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:47:46.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:46.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:47.669 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:48.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:48.671 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:48.671 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:47:48.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:47:48.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:48.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:47:49.150 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 11 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2241527984,v1:127.0.0.1:6811/2241527984] [v2:127.0.0.1:6812/2241527984,v1:127.0.0.1:6813/2241527984] exists,up 7ae69166-b405-4e89-aef6-1a5dce2c00ba 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:47:49.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:47:49.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:47:49.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:47:49.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 2024-09-10T17:47:49.159 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 2024-09-10T17:47:49.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9' 2024-09-10T17:47:49.159 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:47:49.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFhuBmf01CChAAmiRTImHb/DTs3YMsZP2+Tg== 2024-09-10T17:47:49.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFhuBmf01CChAAmiRTImHb/DTs3YMsZP2+Tg=="}' 2024-09-10T17:47:49.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 -i td/test-erasure-eio/2/new.json 2024-09-10T17:47:49.699 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:47:49.710 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:47:49.711 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFhuBmf01CChAAmiRTImHb/DTs3YMsZP2+Tg== --osd-uuid f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 2024-09-10T17:47:49.741 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:49.740+0000 7fe216e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:49.784 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:49.783+0000 7fe216e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:49.785 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:49.784+0000 7fe216e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:49.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:49.785+0000 7fe216e0c640 -1 bdev(0x55b36d333c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:47:49.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:49.785+0000 7fe216e0c640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:47:55.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:47:55.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:47:55.432 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:47:55.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:47:55.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:47:55.964 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:47:55.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:47:55.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:47:55.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:47:55.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:47:55.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:47:55.990 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:55.989+0000 7f5ff0c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:56.004 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:56.002+0000 7f5ff0c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:56.005 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:56.004+0000 7f5ff0c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:47:56.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:56.444 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:47:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:47:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:56.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:47:56.860 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:57.604 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:57.603+0000 7f5ff0c16640 -1 Falling back to public interface 2024-09-10T17:47:57.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:57.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:57.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:47:57.862 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:47:57.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:57.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:47:58.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:47:59.183 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:47:59.182+0000 7f5ff0c16640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:47:59.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:47:59.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:47:59.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:47:59.295 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:47:59.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:47:59.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:47:59.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:00.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:00.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:00.879 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:48:00.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:48:00.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:00.879 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:01.269 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:02.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:02.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:02.271 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:48:02.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:48:02.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:02.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:02.705 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:03.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:03.706 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:03.707 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:48:03.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:48:03.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:03.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:04.159 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:05.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:05.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:05.161 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:48:05.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:48:05.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:05.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:05.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:06.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:06.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:06.616 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:48:06.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:48:06.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:06.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:07.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:08.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:08.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:08.173 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:48:08.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:48:08.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:08.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:08.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:09.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:09.663 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:48:09.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:09.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:48:09.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:09.664 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:10.121 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:11.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:11.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:11.125 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:48:11.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:48:11.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:11.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:11.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:12.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:12.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:12.529 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:48:12.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:48:12.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:12.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:13.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:14.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:14.106 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:48:14.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:14.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:48:14.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:14.106 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:15.457 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:16.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:16.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:16.459 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:48:16.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:48:16.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:16.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:17.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:18.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:18.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:18.382 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:48:18.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:48:18.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:18.382 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:19.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:20.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:20.339 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:20.339 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:48:20.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:48:20.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:20.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:21.466 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:22.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:22.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:22.468 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:48:22.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:48:22.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:22.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:22.869 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:23.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:23.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:23.871 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:48:23.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:48:23.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:23.872 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:48:24.267 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1715049340,v1:127.0.0.1:6819/1715049340] [v2:127.0.0.1:6820/1715049340,v1:127.0.0.1:6821/1715049340] exists,up f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 2024-09-10T17:48:24.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:48:24.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:48:24.272 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:48:24.273 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:48:24.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98' 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98 2024-09-10T17:48:24.275 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:48:24.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBohuBmBts3ERAAkj4pNb+Guu9zRabwrIwcjg== 2024-09-10T17:48:24.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBohuBmBts3ERAAkj4pNb+Guu9zRabwrIwcjg=="}' 2024-09-10T17:48:24.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98 -i td/test-erasure-eio/3/new.json 2024-09-10T17:48:25.202 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:48:25.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:48:25.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBohuBmBts3ERAAkj4pNb+Guu9zRabwrIwcjg== --osd-uuid 8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98 2024-09-10T17:48:25.244 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:25.242+0000 7f2e73211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:25.245 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:25.244+0000 7f2e73211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:25.246 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:25.245+0000 7f2e73211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:25.246+0000 7f2e73211640 -1 bdev(0x556ecd231c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:48:25.247 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:25.246+0000 7f2e73211640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:48:31.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:48:31.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:48:31.470 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:48:31.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:48:31.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:48:32.009 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:48:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:48:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:48:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:48:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:48:32.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:48:32.037 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:32.035+0000 7f361f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:32.044 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:32.043+0000 7f361f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:32.045 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:32.044+0000 7f361f558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:48:32.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:48:32.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:48:32.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:48:32.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:48:32.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:48:32.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:32.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:48:32.403 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:48:32.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:32.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:32.851 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:33.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:33.853 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:33.853 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:48:33.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:48:33.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:33.857 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:33.908 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:33.907+0000 7f361f558640 -1 Falling back to public interface 2024-09-10T17:48:34.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:35.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:35.270 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:35.270 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:48:35.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:48:35.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:35.271 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:35.489 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:48:35.487+0000 7f361f558640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:48:35.688 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:36.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:36.689 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:36.689 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:48:36.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:48:36.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:36.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:37.179 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:38.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:38.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:38.181 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:48:38.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:48:38.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:38.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:38.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:39.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:39.831 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:39.831 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:48:39.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:48:39.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:39.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:40.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:41.336 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:41.337 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:41.337 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:48:41.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:48:41.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:41.338 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:41.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:42.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:42.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:42.735 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:48:42.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:48:42.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:42.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:43.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:44.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:44.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:44.130 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:48:44.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:48:44.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:44.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:44.781 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:45.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:45.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:45.783 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:48:45.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:48:45.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:45.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:46.388 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:47.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:47.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:47.390 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:48:47.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:48:47.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:47.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:47.784 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:48.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:48.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:48.786 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:48:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:48:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:48.786 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:49.483 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:50.485 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:50.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:50.566 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:48:50.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:48:50.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:50.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:51.016 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:52.017 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:52.018 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:48:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:48:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:52.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:53.147 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:54.148 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:54.149 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:48:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:48:54.149 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:54.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:55.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:56.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:56.236 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:48:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:48:56.236 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:56.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:57.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:58.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:58.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:58.038 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:48:58.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:48:58.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:58.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:48:58.989 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:48:59.990 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:48:59.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:48:59.991 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:48:59.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:48:59.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:48:59.992 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:49:00.418 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4271268239,v1:127.0.0.1:6827/4271268239] [v2:127.0.0.1:6828/4271268239,v1:127.0.0.1:6829/4271268239] exists,up 8725ac0a-6bdf-40aa-9c0d-a37a1b0b7e98 2024-09-10T17:49:00.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:49:00.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:49:00.419 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:00.420 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:49:00.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:49:00.421 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:49:00.498 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:49:00.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:49:00.507 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:46:49.979+0000 7f1949fb0640 0 load: jerasure load: lrc 2024-09-10T17:49:00.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:49:00.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:49:01.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:49:01.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:49:02.755 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:49:02.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:49:03.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:49:03.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:49:03.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:49:03.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:49:03.769 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:49:03.770 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:49:03.770 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:49:03.770 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:49:03.770 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:49:03.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:49:03.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:49:03.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:49:03.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:49:03.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:49:03.871 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:49:03.872 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:49:04.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:04.267 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:49:04.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803808 2024-09-10T17:49:04.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803808 2024-09-10T17:49:04.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808' 2024-09-10T17:49:04.379 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:04.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:49:04.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=47244640278 2024-09-10T17:49:04.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 47244640278 2024-09-10T17:49:04.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-47244640278' 2024-09-10T17:49:04.492 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:04.492 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:49:04.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411345 2024-09-10T17:49:04.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411345 2024-09-10T17:49:04.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-47244640278 2-77309411345' 2024-09-10T17:49:04.603 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:04.603 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:49:04.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084294 2024-09-10T17:49:04.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084294 2024-09-10T17:49:04.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803808 1-47244640278 2-77309411345 3-120259084294' 2024-09-10T17:49:04.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:04.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803808 2024-09-10T17:49:04.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:04.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:49:04.716 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803808 2024-09-10T17:49:04.716 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:04.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803808 2024-09-10T17:49:04.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-10T17:49:04.718 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803808 2024-09-10T17:49:04.718 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:05.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-10T17:49:05.097 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:49:06.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:49:06.098 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:06.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-10T17:49:06.520 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:06.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-47244640278 2024-09-10T17:49:06.520 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:06.521 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:49:06.522 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-47244640278 2024-09-10T17:49:06.522 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:06.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=47244640278 2024-09-10T17:49:06.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 47244640278' 2024-09-10T17:49:06.523 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 47244640278 2024-09-10T17:49:06.523 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:49:06.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 47244640278 -lt 47244640278 2024-09-10T17:49:06.951 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:06.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411345 2024-09-10T17:49:06.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:06.952 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:49:06.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411345 2024-09-10T17:49:06.953 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:06.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411345 2024-09-10T17:49:06.954 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 77309411345 2024-09-10T17:49:06.954 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411345' 2024-09-10T17:49:06.954 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:49:07.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411345 -lt 77309411345 2024-09-10T17:49:07.377 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:07.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084294 2024-09-10T17:49:07.378 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:07.378 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:49:07.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084294 2024-09-10T17:49:07.379 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:07.380 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084294 2024-09-10T17:49:07.380 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084294 2024-09-10T17:49:07.381 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-09-10T17:49:07.381 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:49:07.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084295 -lt 120259084294 2024-09-10T17:49:07.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:49:07.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:07.796 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:08.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:49:08.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:49:08.319 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:49:08.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:49:08.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:49:08.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:49:08.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:49:08.320 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:49:08.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:49:08.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:49:08.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:08.698 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:09.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:49:09.215 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_subread_missing: local shard_id=1 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-10T17:49:09.216 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-86744-1 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-86744-1 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-86744-1 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:49:09.217 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:49:09.218 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-86744-1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:49:09.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-86744-1 td/test-erasure-eio 1 2024-09-10T17:49:09.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-10T17:49:09.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-10T17:49:09.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-86744-1 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-86744-1 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-86744-1 2024-09-10T17:49:09.399 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-86744-1 2024-09-10T17:49:09.400 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:49:09.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:49:09.836 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-86744-1 remove 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-86744-1 remove 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:49:09.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:09.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:10.142 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-86744-1 remove 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-10T17:49:10.143 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-86744-1 remove 2024-09-10T17:49:12.037 INFO:tasks.workunit.client.0.smithi086.stdout:remove 1#2:72a937ed:::obj-remove-86744-1:head# 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:49:12.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:12.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:49:12.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.1 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T17:49:12.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:49:12.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:49:12.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:49:12.593 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:12.592+0000 7fc2f4158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:12.595 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:12.594+0000 7fc2f4158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:12.597 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:12.596+0000 7fc2f4158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:12.991 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:13.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:13.972 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:13.971+0000 7fc2f4158640 -1 Falling back to public interface 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:14.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:14.852 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:15.629 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:15.628+0000 7fc2f4158640 -1 osd.1 37 log_to_monitors true 2024-09-10T17:49:15.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:15.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:15.854 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:49:15.854 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:49:15.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:15.855 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:osd.1 up in weight 1 up_from 42 up_thru 19 down_at 38 last_clean_interval [11,37) [v2:127.0.0.1:6810/3491504138,v1:127.0.0.1:6811/3491504138] [v2:127.0.0.1:6812/3491504138,v1:127.0.0.1:6813/3491504138] exists,up 7ae69166-b405-4e89-aef6-1a5dce2c00ba 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:49:16.331 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:49:16.332 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:49:16.332 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:49:16.332 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:49:16.332 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:49:16.332 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:49:16.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:49:16.333 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:49:16.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:49:16.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:49:16.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:49:16.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:49:16.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:49:16.423 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:49:17.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:49:17.113 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:17.113 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:49:17.114 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:49:17.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:49:17.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:17.114 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:49:17.430 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803812 2024-09-10T17:49:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803812 2024-09-10T17:49:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812' 2024-09-10T17:49:17.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:17.431 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:49:17.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626435 2024-09-10T17:49:17.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626435 2024-09-10T17:49:17.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-180388626435' 2024-09-10T17:49:17.583 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:17.583 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:49:17.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=77309411349 2024-09-10T17:49:17.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 77309411349 2024-09-10T17:49:17.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-180388626435 2-77309411349' 2024-09-10T17:49:17.752 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:17.752 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084299 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084299 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803812 1-180388626435 2-77309411349 3-120259084299' 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803812 2024-09-10T17:49:17.876 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:17.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:49:17.877 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803812 2024-09-10T17:49:17.877 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:17.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803812 2024-09-10T17:49:17.878 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-10T17:49:17.878 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803812 2024-09-10T17:49:17.878 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:18.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-10T17:49:18.422 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:18.422 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-180388626435 2024-09-10T17:49:18.423 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:18.423 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:49:18.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-180388626435 2024-09-10T17:49:18.424 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:18.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626435 2024-09-10T17:49:18.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 180388626435' 2024-09-10T17:49:18.425 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 180388626435 2024-09-10T17:49:18.425 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:49:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-10T17:49:18.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:18.804 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-77309411349 2024-09-10T17:49:18.804 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:18.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:49:18.806 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-77309411349 2024-09-10T17:49:18.806 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:18.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=77309411349 2024-09-10T17:49:18.807 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 77309411349' 2024-09-10T17:49:18.807 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 77309411349 2024-09-10T17:49:18.807 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:49:19.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 77309411349 -lt 77309411349 2024-09-10T17:49:19.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:19.242 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084299 2024-09-10T17:49:19.243 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:19.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:49:19.243 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084299 2024-09-10T17:49:19.243 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:19.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084299 2024-09-10T17:49:19.245 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084299' 2024-09-10T17:49:19.245 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 120259084299 2024-09-10T17:49:19.245 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:49:19.670 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084299 -lt 120259084299 2024-09-10T17:49:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:49:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:19.670 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:49:20.195 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:49:20.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:49:20.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:49:20.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:20.566 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-86744-1 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:49:21.090 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-86744-1 2024-09-10T17:49:21.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:49:21.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:49:21.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-86744-1 td/test-erasure-eio/COPY 2024-09-10T17:49:21.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:49:21.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:49:21.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-10T17:49:21.259 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-10T17:49:21.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-86744-1 td/test-erasure-eio 2 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-86744-1 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-10T17:49:21.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-86744-1 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-86744-1 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-86744-1 2024-09-10T17:49:21.261 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-86744-1 remove 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2024-09-10T17:49:21.574 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-86744-1 remove 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:21.575 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-86744-1 remove 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T17:49:21.880 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-10T17:49:21.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-86744-1 remove 2024-09-10T17:49:23.815 INFO:tasks.workunit.client.0.smithi086.stdout:remove 2#2:72a937ed:::obj-remove-86744-1:head# 2024-09-10T17:49:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-10T17:49:24.341 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:49:24.342 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:49:24.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:49:24.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:49:24.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:49:24.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:49:24.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:49:24.344 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:start osd.2 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:49:24.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:49:24.373 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:24.371+0000 7f1358412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:24.375 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:24.374+0000 7f1358412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:24.376 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:24.375+0000 7f1358412640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:24.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:49:24.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:0 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:24.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:49:25.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:26.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:49:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:49:26.208 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:26.207+0000 7f1358412640 -1 Falling back to public interface 2024-09-10T17:49:26.566 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:27.567 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:49:27.786 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:27.784+0000 7f1358412640 -1 osd.2 45 log_to_monitors true 2024-09-10T17:49:27.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:28.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:28.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:28.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:49:28.981 INFO:tasks.workunit.client.0.smithi086.stderr:3 2024-09-10T17:49:28.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:28.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 up in weight 1 up_from 50 up_thru 38 down_at 46 last_clean_interval [18,45) [v2:127.0.0.1:6818/3266882951,v1:127.0.0.1:6819/3266882951] [v2:127.0.0.1:6820/3266882951,v1:127.0.0.1:6821/3266882951] exists,up f88b4933-ce0e-40a6-b2c4-ddd9f7cbbcc9 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:49:29.453 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:49:29.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:49:29.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:49:29.454 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:49:29.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:49:29.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:49:29.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:49:29.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:49:29.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:49:29.543 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:49:29.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:49:29.964 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:49:29.964 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:49:29.965 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:49:29.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:49:29.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:29.965 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:49:30.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803817 2024-09-10T17:49:30.077 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803817 2024-09-10T17:49:30.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817' 2024-09-10T17:49:30.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:30.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:49:30.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626439 2024-09-10T17:49:30.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626439 2024-09-10T17:49:30.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-180388626439' 2024-09-10T17:49:30.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:30.190 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:49:30.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-10T17:49:30.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-10T17:49:30.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-180388626439 2-214748364803' 2024-09-10T17:49:30.340 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:49:30.340 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:49:30.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084304 2024-09-10T17:49:30.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084304 2024-09-10T17:49:30.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803817 1-180388626439 2-214748364803 3-120259084304' 2024-09-10T17:49:30.452 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:30.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803817 2024-09-10T17:49:30.452 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:30.453 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:49:30.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803817 2024-09-10T17:49:30.453 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:30.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803817 2024-09-10T17:49:30.455 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-10T17:49:30.455 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.0 seq 25769803817 2024-09-10T17:49:30.455 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:30.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803815 -lt 25769803817 2024-09-10T17:49:30.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:49:31.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:49:31.841 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:49:32.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803817 -lt 25769803817 2024-09-10T17:49:32.232 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:32.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-180388626439 2024-09-10T17:49:32.232 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:32.233 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:49:32.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-180388626439 2024-09-10T17:49:32.233 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:32.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626439 2024-09-10T17:49:32.235 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 180388626439' 2024-09-10T17:49:32.235 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.1 seq 180388626439 2024-09-10T17:49:32.235 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:49:32.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626440 -lt 180388626439 2024-09-10T17:49:32.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:32.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-10T17:49:32.661 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:32.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:49:32.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-10T17:49:32.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:32.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-10T17:49:32.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-10T17:49:32.663 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.2 seq 214748364803 2024-09-10T17:49:32.664 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:49:33.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-10T17:49:33.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:49:33.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084304 2024-09-10T17:49:33.078 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:49:33.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:49:33.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084304 2024-09-10T17:49:33.080 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:49:33.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084304 2024-09-10T17:49:33.081 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084304' 2024-09-10T17:49:33.081 INFO:tasks.workunit.client.0.smithi086.stderr:waiting osd.3 seq 120259084304 2024-09-10T17:49:33.081 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:49:33.504 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084304 -lt 120259084304 2024-09-10T17:49:33.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:49:33.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:33.505 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:49:34.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:49:34.425 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:49:34.426 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:49:34.426 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:49:34.426 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-86744-1 2024-09-10T17:49:34.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:49:34.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:49:34.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-86744-1 2024-09-10T17:49:34.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:49:34.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:49:34.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-86744-1 td/test-erasure-eio/COPY 2024-09-10T17:49:35.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:49:35.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:49:35.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-10T17:49:35.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:49:35.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:49:35.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:49:35.487 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:49:35.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:49:36.050 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:49:36.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:49:36.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:49:36.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:49:36.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:49:36.062 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:36.063 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:36.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:36.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:36.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:36.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:36.171 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:49:36.172 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:49:36.172 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:49:36.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:49:36.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:49:36.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:49:36.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:36.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:49:36.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:49:36.176 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:36.176 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:49:36.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:49:36.177 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:49:36.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:49:36.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.198 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.198 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:49:36.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:49:36.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:49:36.202 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:49:36.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:49:36.203 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:49:36.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:49:36.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:49:36.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:49:36.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:36.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:49:36.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:49:36.206 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:49:36.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:49:36.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:49:36.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:49:36.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:49:36.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.209 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:49:36.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:49:36.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:49:36.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:49:36.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:49:36.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.211 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:49:36.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:49:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:49:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:49:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:49:36.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:49:36.214 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:36.675 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:49:36.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:49:36.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:49:36.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:49:36.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:49:36.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:49:36.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:49:36.869 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:49:36.870 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:36.950 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:36.951 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:36.951 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:49:36.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:49:36.951 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:49:37.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:49:37.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:49:37.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:49:37.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:37.874 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:37.874 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:37.875 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:37.875 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:37.875 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:37.875 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:49:37.875 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:49:37.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:49:37.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:49:38.200 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:49:38.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:49:39.213 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:49:39.286 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:49:39.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:49:39.295 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:49:36.707+0000 7f699611ec80 0 load: jerasure load: lrc 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:351: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:353: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:49:39.296 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:49:39.297 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:49:39.301 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:39.302 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:49:39.303 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:49:39.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a 2024-09-10T17:49:39.304 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a 2024-09-10T17:49:39.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a' 2024-09-10T17:49:39.305 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:49:39.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzhuBmh1YBExAA2+TuWQpEOJBwPAieCZ1cAg== 2024-09-10T17:49:39.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzhuBmh1YBExAA2+TuWQpEOJBwPAieCZ1cAg=="}' 2024-09-10T17:49:39.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a -i td/test-erasure-eio/0/new.json 2024-09-10T17:49:39.577 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:49:39.586 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:49:39.587 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzhuBmh1YBExAA2+TuWQpEOJBwPAieCZ1cAg== --osd-uuid bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a 2024-09-10T17:49:39.616 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:39.615+0000 7f498c616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:39.654 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:39.653+0000 7f498c616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:39.655 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:39.655+0000 7f498c616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:39.657 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:39.655+0000 7f498c616640 -1 bdev(0x55f13a773c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:49:39.657 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:39.655+0000 7f498c616640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:49:45.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:49:45.896 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:49:45.897 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:49:45.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:49:45.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:49:46.431 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:49:46.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:49:46.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:49:46.432 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:49:46.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:49:46.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:49:46.459 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:46.457+0000 7f6802b2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:46.476 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:46.475+0000 7f6802b2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:46.477 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:46.476+0000 7f6802b2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:49:46.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:49:46.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:49:46.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:49:46.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:49:46.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:49:46.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:46.908 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:49:46.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:49:46.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:46.908 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:47.333 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:47.597 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:47.596+0000 7f6802b2c640 -1 Falling back to public interface 2024-09-10T17:49:48.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:48.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:48.334 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:49:48.335 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:49:48.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:48.335 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:48.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:49.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:49:49.468+0000 7f6802b2c640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:49:49.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:49.760 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:49.760 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:49:49.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:49:49.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:49.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:50.207 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:51.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:51.208 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:51.209 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:49:51.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:49:51.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:51.209 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:51.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:52.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:52.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:52.627 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:49:52.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:49:52.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:52.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:53.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:54.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:54.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:54.051 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:49:54.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:49:54.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:54.052 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:54.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:55.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:55.560 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:55.560 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:49:55.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:49:55.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:55.561 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:56.003 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:57.004 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:57.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:57.005 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:49:57.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:49:57.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:57.005 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:57.493 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:58.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:58.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:58.496 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:49:58.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:49:58.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:58.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:49:58.909 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:49:59.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:49:59.911 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:49:59.911 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:49:59.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:49:59.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:49:59.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:00.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:01.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:01.366 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:01.366 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:50:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:50:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:01.367 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:02.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:03.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:03.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:03.110 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:50:03.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:50:03.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:03.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:03.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:04.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:04.839 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:04.839 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:50:04.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:50:04.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:04.840 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:05.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:06.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:06.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:06.635 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:50:06.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:50:06.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:06.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:07.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:08.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:08.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:08.888 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:50:08.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:50:08.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:08.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:09.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:10.769 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:10.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:10.770 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:50:10.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:50:10.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:10.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:11.171 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:12.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:12.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:12.173 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:50:12.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:50:12.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:12.174 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:50:12.686 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/78349353,v1:127.0.0.1:6803/78349353] [v2:127.0.0.1:6804/78349353,v1:127.0.0.1:6805/78349353] exists,up bdb2ef61-6e6f-4cac-b90e-a0d7e4486c7a 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:50:12.690 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:12.691 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:50:12.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4e1aa12c-4970-4d23-bdb5-8b5f4f72f304 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4e1aa12c-4970-4d23-bdb5-8b5f4f72f304' 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 4e1aa12c-4970-4d23-bdb5-8b5f4f72f304 2024-09-10T17:50:12.693 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:50:12.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUhuBmAhkpKhAAF7mQfzyar034NovmYEIdQA== 2024-09-10T17:50:12.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUhuBmAhkpKhAAF7mQfzyar034NovmYEIdQA=="}' 2024-09-10T17:50:12.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4e1aa12c-4970-4d23-bdb5-8b5f4f72f304 -i td/test-erasure-eio/1/new.json 2024-09-10T17:50:13.386 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:50:13.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:50:13.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUhuBmAhkpKhAAF7mQfzyar034NovmYEIdQA== --osd-uuid 4e1aa12c-4970-4d23-bdb5-8b5f4f72f304 2024-09-10T17:50:13.428 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:13.426+0000 7fd0a4aef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:13.445 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:13.444+0000 7fd0a4aef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:13.446 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:13.445+0000 7fd0a4aef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:13.447 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:13.446+0000 7fd0a4aef640 -1 bdev(0x5562ca823c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:50:13.447 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:13.446+0000 7fd0a4aef640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:50:19.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:50:19.237 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:50:19.238 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:50:19.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:50:19.238 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:50:19.771 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:50:19.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:50:19.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:50:19.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:50:19.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:50:19.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:50:19.799 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:19.798+0000 7f3ab9151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:19.824 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:19.822+0000 7f3ab9151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:19.825 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:19.824+0000 7f3ab9151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:20.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:50:20.210 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:50:20.211 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:20.212 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:20.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:21.175 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:21.174+0000 7f3ab9151640 -1 Falling back to public interface 2024-09-10T17:50:21.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:21.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:21.632 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:50:21.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:50:21.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:21.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:22.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:22.768 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:22.767+0000 7f3ab9151640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:50:23.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:23.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:23.020 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:50:23.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:50:23.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:23.021 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:23.570 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:24.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:24.571 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:24.572 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:50:24.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:50:24.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:24.572 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:24.966 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:25.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:25.967 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:25.968 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:50:25.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:50:25.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:25.968 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:26.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:27.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:27.475 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:27.476 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:50:27.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:50:27.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:27.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:27.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:28.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:28.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:28.922 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:50:28.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:50:28.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:28.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:29.316 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:30.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:30.318 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:30.318 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:50:30.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:50:30.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:30.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:30.741 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:31.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:31.742 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:31.742 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:50:31.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:50:31.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:31.743 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:32.137 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:33.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:33.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:33.139 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:50:33.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:50:33.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:33.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:33.598 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:34.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:34.600 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:34.600 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:50:34.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:50:34.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:34.601 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:35.274 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:36.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:36.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:36.276 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:50:36.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:50:36.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:36.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:37.093 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:38.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:38.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:38.095 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:50:38.095 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:50:38.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:38.096 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:38.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:39.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:39.847 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:39.847 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:50:39.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:50:39.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:39.848 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:41.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:42.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:42.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:42.010 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:50:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:50:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:42.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:42.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:43.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:43.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:43.922 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:50:43.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:50:43.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:43.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:44.881 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:45.882 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:45.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:45.883 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:50:45.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:50:45.883 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:45.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:50:46.401 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/939833820,v1:127.0.0.1:6811/939833820] [v2:127.0.0.1:6812/939833820,v1:127.0.0.1:6813/939833820] exists,up 4e1aa12c-4970-4d23-bdb5-8b5f4f72f304 2024-09-10T17:50:46.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:50:46.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:50:46.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:50:46.406 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:50:46.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:50:46.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:50:46.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:50:46.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:50:46.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:50:46.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22f88940-f741-4c32-a847-acda420d0598 2024-09-10T17:50:46.410 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 22f88940-f741-4c32-a847-acda420d0598 2024-09-10T17:50:46.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 22f88940-f741-4c32-a847-acda420d0598' 2024-09-10T17:50:46.410 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:50:46.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2huBmt701GRAAPLZkSrgw6SpXHHAfXrGwKQ== 2024-09-10T17:50:46.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2huBmt701GRAAPLZkSrgw6SpXHHAfXrGwKQ=="}' 2024-09-10T17:50:46.424 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22f88940-f741-4c32-a847-acda420d0598 -i td/test-erasure-eio/2/new.json 2024-09-10T17:50:47.280 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:50:47.292 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:50:47.293 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2huBmt701GRAAPLZkSrgw6SpXHHAfXrGwKQ== --osd-uuid 22f88940-f741-4c32-a847-acda420d0598 2024-09-10T17:50:47.322 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:47.321+0000 7fdd1b4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:47.324 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:47.323+0000 7fdd1b4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:47.325 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:47.324+0000 7fdd1b4f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:47.325 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:47.324+0000 7fdd1b4f1640 -1 bdev(0x55af52d8bc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:50:47.325 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:47.324+0000 7fdd1b4f1640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:50:53.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:50:53.356 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:50:53.357 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:50:53.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:50:53.357 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:50:53.891 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:50:53.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:50:53.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:50:53.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:50:53.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:50:53.892 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:50:53.919 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:53.918+0000 7f436131d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:53.934 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:53.933+0000 7f436131d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:53.935 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:53.934+0000 7f436131d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:50:54.321 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:50:54.322 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:50:54.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:54.323 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:50:54.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:55.537 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:55.536+0000 7f436131d640 -1 Falling back to public interface 2024-09-10T17:50:55.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:55.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:55.734 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:50:55.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:50:55.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:55.735 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:50:56.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:57.104 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:50:57.103+0000 7f436131d640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:50:57.133 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:57.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:57.134 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:50:57.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:50:57.134 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:57.135 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:50:57.773 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:50:58.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:50:58.774 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:50:58.774 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:50:58.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:50:58.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:50:58.775 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:50:59.188 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:00.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:00.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:00.190 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:51:00.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:51:00.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:00.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:00.582 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:01.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:01.584 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:01.584 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:51:01.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:51:01.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:01.585 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:02.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:03.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:03.048 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:51:03.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:03.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:51:03.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:03.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:03.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:04.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:04.446 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:51:04.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:51:04.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:04.447 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:04.969 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:05.971 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:05.971 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:51:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:51:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:05.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:06.409 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:07.410 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:07.410 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:51:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:51:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:07.411 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:07.931 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:08.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:08.933 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:08.933 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:51:08.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:51:08.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:08.934 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:09.343 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:10.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:10.345 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:10.345 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:51:10.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:51:10.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:10.346 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:11.648 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:12.650 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:12.650 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:51:12.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:12.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:51:12.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:12.651 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:13.435 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:14.436 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:14.437 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:14.437 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:51:14.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:51:14.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:14.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:14.959 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:15.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:15.961 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:15.962 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:51:15.962 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:51:15.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:15.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:16.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:17.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:17.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:17.478 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:51:17.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:51:17.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:17.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:18.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:19.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:19.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:19.158 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:51:19.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:51:19.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:19.158 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:20.018 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:21.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:21.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:21.020 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:51:21.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:51:21.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:21.020 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:51:21.611 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1704750229,v1:127.0.0.1:6819/1704750229] [v2:127.0.0.1:6820/1704750229,v1:127.0.0.1:6821/1704750229] exists,up 22f88940-f741-4c32-a847-acda420d0598 2024-09-10T17:51:21.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:51:21.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:51:21.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:51:21.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:51:21.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:51:21.616 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:51:21.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:51:21.617 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:51:21.617 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:51:21.620 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=223a06ce-05f8-4445-a6fc-b1805f0527ee 2024-09-10T17:51:21.621 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 223a06ce-05f8-4445-a6fc-b1805f0527ee' 2024-09-10T17:51:21.622 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 223a06ce-05f8-4445-a6fc-b1805f0527ee 2024-09-10T17:51:21.622 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:51:21.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZh+BmXRnDJRAAaHqzT88O2wfJzvAXThor6Q== 2024-09-10T17:51:21.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZh+BmXRnDJRAAaHqzT88O2wfJzvAXThor6Q=="}' 2024-09-10T17:51:21.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 223a06ce-05f8-4445-a6fc-b1805f0527ee -i td/test-erasure-eio/3/new.json 2024-09-10T17:51:22.144 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:51:22.155 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:51:22.156 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZh+BmXRnDJRAAaHqzT88O2wfJzvAXThor6Q== --osd-uuid 223a06ce-05f8-4445-a6fc-b1805f0527ee 2024-09-10T17:51:22.186 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:22.185+0000 7f4059334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:22.255 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:22.254+0000 7f4059334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:22.256 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:22.255+0000 7f4059334640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:22.256 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:22.255+0000 7f4059334640 -1 bdev(0x561225ea1c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:51:22.256 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:22.255+0000 7f4059334640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:51:28.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:51:28.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:51:28.244 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:51:28.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:51:28.244 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:51:28.781 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:51:28.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:51:28.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:51:28.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:51:28.782 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:51:28.783 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:51:28.807 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:28.805+0000 7f52cd611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:28.827 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:28.826+0000 7f52cd611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:28.828 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:28.827+0000 7f52cd611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:51:29.234 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:51:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:51:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:51:29.241 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:51:29.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:51:29.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:29.242 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:51:29.242 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:51:29.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:29.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:29.612 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:30.176 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:30.175+0000 7f52cd611640 -1 Falling back to public interface 2024-09-10T17:51:30.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:30.614 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:30.614 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:51:30.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:51:30.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:30.615 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:31.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:32.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:32.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:32.038 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:51:32.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:51:32.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:32.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:32.097 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:51:32.096+0000 7f52cd611640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:51:32.642 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:33.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:33.644 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:33.644 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:51:33.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:51:33.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:33.645 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:34.350 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:35.351 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:35.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:35.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:51:35.352 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:51:35.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:35.353 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:35.755 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:36.756 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:36.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:36.757 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:51:36.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:51:36.757 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:36.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:37.257 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:38.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:38.258 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:38.259 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:51:38.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:51:38.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:38.259 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:38.657 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:39.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:39.659 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:39.659 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:51:39.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:51:39.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:39.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:40.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:41.131 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:41.131 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:51:41.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:41.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:51:41.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:41.132 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:41.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:42.530 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:42.531 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:42.531 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:51:42.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:51:42.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:42.532 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:42.936 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:43.937 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:51:43.938 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:51:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:43.938 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:44.426 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:45.427 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:45.428 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:45.428 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:45.429 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:45.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:46.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:46.886 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:51:46.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:46.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:51:46.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:46.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:47.631 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:48.632 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:48.633 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:51:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:51:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:48.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:49.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:50.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:50.403 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:50.403 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:51:50.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:51:50.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:50.404 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:51.178 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:52.180 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:52.180 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:51:52.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:52.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:51:52.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:52.181 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:52.947 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:53.948 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:53.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:53.949 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:51:53.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:51:53.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:53.949 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:54.793 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:55.794 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:55.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:55.795 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:51:55.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:51:55.795 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:55.796 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:56.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:51:57.416 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:51:57.416 INFO:tasks.workunit.client.0.smithi086.stdout:18 2024-09-10T17:51:57.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:51:57.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T17:51:57.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:51:57.417 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:51:58.138 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3431079274,v1:127.0.0.1:6827/3431079274] [v2:127.0.0.1:6828/3431079274,v1:127.0.0.1:6829/3431079274] exists,up 223a06ce-05f8-4445-a6fc-b1805f0527ee 2024-09-10T17:51:58.139 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:51:58.140 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:51:58.219 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:51:58.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:51:58.229 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:49:47.600+0000 7f6802b2c640 0 load: jerasure load: lrc 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:355: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:356: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:51:58.231 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:51:59.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:51:59.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:52:00.051 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:52:00.062 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:52:01.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:52:01.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:52:01.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:52:01.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:52:01.065 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:52:01.065 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:52:01.066 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:52:01.066 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:52:01.066 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:52:01.066 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:52:01.163 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:52:01.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:52:01.590 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:52:01.590 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:52:01.590 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:52:01.590 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:52:01.591 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:01.591 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:52:01.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803802 2024-09-10T17:52:01.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803802 2024-09-10T17:52:01.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803802' 2024-09-10T17:52:01.701 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:01.701 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:52:01.812 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574868 2024-09-10T17:52:01.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574868 2024-09-10T17:52:01.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803802 1-55834574868' 2024-09-10T17:52:01.813 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:01.813 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:52:01.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378639 2024-09-10T17:52:01.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378639 2024-09-10T17:52:01.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803802 1-55834574868 2-81604378639' 2024-09-10T17:52:01.925 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:01.926 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:52:02.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084300 2024-09-10T17:52:02.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084300 2024-09-10T17:52:02.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803802 1-55834574868 2-81604378639 3-120259084300' 2024-09-10T17:52:02.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:02.039 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803802 2024-09-10T17:52:02.040 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:02.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:52:02.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803802 2024-09-10T17:52:02.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:02.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803802 2024-09-10T17:52:02.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-09-10T17:52:02.042 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803802 2024-09-10T17:52:02.042 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:52:02.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803802 -lt 25769803802 2024-09-10T17:52:02.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:02.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574868 2024-09-10T17:52:02.461 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:02.462 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:52:02.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574868 2024-09-10T17:52:02.462 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:02.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574868 2024-09-10T17:52:02.463 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-09-10T17:52:02.464 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574868 2024-09-10T17:52:02.464 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:52:02.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574868 -lt 55834574868 2024-09-10T17:52:02.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:02.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378639 2024-09-10T17:52:02.887 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:02.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:52:02.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378639 2024-09-10T17:52:02.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:02.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378639 2024-09-10T17:52:02.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-09-10T17:52:02.890 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378639 2024-09-10T17:52:02.890 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:52:03.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378637 -lt 81604378639 2024-09-10T17:52:03.304 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:52:04.305 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:52:04.306 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:52:04.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378639 -lt 81604378639 2024-09-10T17:52:04.731 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:04.732 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084300 2024-09-10T17:52:04.732 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:04.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:52:04.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084300 2024-09-10T17:52:04.733 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:04.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084300 2024-09-10T17:52:04.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084300' 2024-09-10T17:52:04.735 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084300 2024-09-10T17:52:04.735 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:52:05.157 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084301 -lt 120259084300 2024-09-10T17:52:05.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:52:05.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:52:05.158 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:52:05.677 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:52:06.079 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:52:06.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:52:06.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:52:06.079 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-10T17:52:06.593 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-86744-0 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-86744-0 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-86744-0 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:52:06.594 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:52:06.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:52:06.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:52:06.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:52:06.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:52:06.595 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-86744-0 td/test-erasure-eio/ORIGINAL 2024-09-10T17:52:06.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-0 td/test-erasure-eio 0 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-0 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:52:06.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-0 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-0 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-0 2024-09-10T17:52:06.763 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:52:07.172 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:52:07.172 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=3 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:52:07.173 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/3/type 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:52:07.175 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:52:07.176 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:07.176 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:07.176 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:07.176 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:52:07.176 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:52:07.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:52:07.260 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:52:07.261 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.3 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.3.asok 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:52:07.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-86744-0 0 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-0 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-0 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:52:07.348 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:52:07.349 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-86744-0 td/test-erasure-eio/COPY 2024-09-10T17:52:07.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:52:07.386 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:52:07.387 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-10T17:52:07.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-86744-0 2024-09-10T17:52:07.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:52:07.388 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-0 2024-09-10T17:52:07.389 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-0 2024-09-10T17:52:07.389 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-10T17:52:07.771 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:52:07.772 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:07.772 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:07.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:07.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:07.772 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:08.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:08.078 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-10T17:52:08.749 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already out. 2024-09-10T17:52:08.761 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-86744-0 2024-09-10T17:52:08.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-10T17:52:08.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:52:08.762 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-0 2024-09-10T17:52:08.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-0 2024-09-10T17:52:08.762 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:52:09.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:52:09.300 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:52:09.300 INFO:tasks.workunit.client.0.smithi086.stderr:0' 2024-09-10T17:52:09.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-10T17:52:09.300 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-10T17:52:10.108 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already in. 2024-09-10T17:52:10.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-10T17:52:10.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:52:10.122 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:52:10.123 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:52:10.124 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:52:10.125 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:52:10.126 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:52:10.149 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:10.148+0000 7f3e9a00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:10.311 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:10.309+0000 7f3e9a00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:10.312 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:10.311+0000 7f3e9a00b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:10.683 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:10.684 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:52:10.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:52:10.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:10.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:52:11.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:11.666 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:11.665+0000 7f3e9a00b640 -1 Falling back to public interface 2024-09-10T17:52:12.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:12.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:12.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:52:12.204 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:52:12.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:12.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:52:12.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:13.609 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:13.609 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:52:13.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:52:13.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:13.610 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:52:13.872 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:13.870+0000 7f3e9a00b640 -1 osd.2 37 log_to_monitors true 2024-09-10T17:52:14.068 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:15.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:15.069 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:15.069 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:52:15.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:52:15.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:15.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:52:15.660 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 48 up_thru 19 down_at 38 last_clean_interval [19,37) [v2:127.0.0.1:6818/2625125571,v1:127.0.0.1:6819/2625125571] [v2:127.0.0.1:6820/2625125571,v1:127.0.0.1:6821/2625125571] exists,up 22f88940-f741-4c32-a847-acda420d0598 2024-09-10T17:52:15.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:52:15.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:52:15.661 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:52:15.662 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:52:15.751 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:52:16.431 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:16.432 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:52:16.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803807 2024-09-10T17:52:16.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803807 2024-09-10T17:52:16.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807' 2024-09-10T17:52:16.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:16.717 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:52:16.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-10T17:52:16.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-10T17:52:16.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574873' 2024-09-10T17:52:16.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:16.861 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:52:17.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-10T17:52:17.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-10T17:52:17.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574873 2-206158430211' 2024-09-10T17:52:17.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:52:17.113 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084305 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084305 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803807 1-55834574873 2-206158430211 3-120259084305' 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803807 2024-09-10T17:52:17.225 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:17.226 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:52:17.227 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803807 2024-09-10T17:52:17.227 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:17.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803807 2024-09-10T17:52:17.228 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-10T17:52:17.228 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803807 2024-09-10T17:52:17.228 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:52:17.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803807 2024-09-10T17:52:17.639 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:52:18.641 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:52:18.641 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:52:19.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803807 -lt 25769803807 2024-09-10T17:52:19.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:19.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-10T17:52:19.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:19.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:52:19.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-10T17:52:19.028 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:19.029 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-10T17:52:19.029 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574873 2024-09-10T17:52:19.030 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-10T17:52:19.030 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:52:19.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574874 -lt 55834574873 2024-09-10T17:52:19.405 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:19.406 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-206158430211 2024-09-10T17:52:19.406 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:19.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:52:19.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-206158430211 2024-09-10T17:52:19.407 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:19.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-10T17:52:19.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-09-10T17:52:19.409 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 206158430211 2024-09-10T17:52:19.409 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:52:19.820 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-10T17:52:19.821 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:52:19.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-120259084305 2024-09-10T17:52:19.821 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:52:19.822 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:52:19.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-120259084305 2024-09-10T17:52:19.823 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:52:19.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084305 2024-09-10T17:52:19.824 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 120259084305' 2024-09-10T17:52:19.824 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 120259084305 2024-09-10T17:52:19.824 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:52:20.219 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084306 -lt 120259084305 2024-09-10T17:52:20.219 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:52:20.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:52:20.220 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:52:20.736 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:52:21.146 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:52:21.147 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:52:21.147 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:52:21.147 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:52:21.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:52:21.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:52:21.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:52:21.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:52:21.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:52:21.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:52:22.264 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:52:22.275 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:52:22.885 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:22.898 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:23.006 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:23.007 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:23.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:23.008 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:23.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:23.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:23.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:23.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:23.010 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:23.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:23.011 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:23.011 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:23.012 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:23.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:52:23.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:23.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.033 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.033 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:52:23.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:23.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:23.035 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:52:23.036 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:52:23.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:52:23.037 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:52:23.037 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:52:23.038 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:52:23.038 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:52:23.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:52:23.039 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:52:23.039 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:52:23.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:23.040 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:52:23.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:52:23.041 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:52:23.041 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:52:23.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:52:23.042 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:52:23.044 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:52:23.044 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.044 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:52:23.044 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:52:23.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:52:23.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2024-09-10T17:52:23.049 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T17:52:23.049 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.86744 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2024-09-10T17:52:23.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-10T17:52:23.500 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:23.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T17:52:23.685 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T17:52:23.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T17:52:23.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:23.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:23.686 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:23.687 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.688 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.688 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:52:23.688 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:23.688 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get fsid 2024-09-10T17:52:23.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:52:23.768 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T17:52:23.769 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.86744/ceph-mon.a.asok config get mon_host 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2024-09-10T17:52:23.846 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T17:52:24.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T17:52:24.691 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:24.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T17:52:24.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-10T17:52:24.720 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-10T17:52:25.009 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'rbd' already exists 2024-09-10T17:52:25.019 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:52:26.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-10T17:52:26.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T17:52:26.021 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-mon.a.asok 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-10T17:52:26.022 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-mon.a.asok log flush 2024-09-10T17:52:26.094 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:52:26.101 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-10T17:52:26.102 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:52:23.531+0000 7f241f412c80 0 load: jerasure load: lrc 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-10T17:52:26.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-10T17:52:26.104 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-10T17:52:26.104 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:52:26.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:26.108 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:52:26.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:52:26.110 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-10T17:52:26.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:52:26.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f582df02-ae09-423f-bb1f-90d4cf62161c 2024-09-10T17:52:26.111 INFO:tasks.workunit.client.0.smithi086.stdout:add osd0 f582df02-ae09-423f-bb1f-90d4cf62161c 2024-09-10T17:52:26.111 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f582df02-ae09-423f-bb1f-90d4cf62161c' 2024-09-10T17:52:26.111 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:52:26.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBah+BmS9yHBxAAGAEIRbqTnMXAEqanCiNqfw== 2024-09-10T17:52:26.127 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBah+BmS9yHBxAAGAEIRbqTnMXAEqanCiNqfw=="}' 2024-09-10T17:52:26.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f582df02-ae09-423f-bb1f-90d4cf62161c -i td/test-erasure-eio/0/new.json 2024-09-10T17:52:26.386 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:52:26.395 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-10T17:52:26.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBah+BmS9yHBxAAGAEIRbqTnMXAEqanCiNqfw== --osd-uuid f582df02-ae09-423f-bb1f-90d4cf62161c 2024-09-10T17:52:26.428 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:26.426+0000 7f773fe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:26.455 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:26.453+0000 7f773fe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:26.455 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:26.455+0000 7f773fe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:26.456 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:26.455+0000 7f773fe12640 -1 bdev(0x56330de95c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2024-09-10T17:52:26.456 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:26.455+0000 7f773fe12640 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-10T17:52:32.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-10T17:52:32.876 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:52:32.877 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd0 key to auth repository 2024-09-10T17:52:32.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T17:52:32.877 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:52:33.407 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.0 2024-09-10T17:52:33.407 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T17:52:33.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:52:33.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:52:33.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:52:33.408 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:52:33.434 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:33.433+0000 7f1f2e824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:33.459 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:33.458+0000 7f1f2e824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:33.460 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:33.459+0000 7f1f2e824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:52:33.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:52:33.865 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:52:33.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:33.866 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:34.282 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:34.813 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:34.812+0000 7f1f2e824640 -1 Falling back to public interface 2024-09-10T17:52:35.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:35.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:35.283 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:52:35.284 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:52:35.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:35.284 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:35.716 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:36.395 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:52:36.393+0000 7f1f2e824640 -1 osd.0 0 log_to_monitors true 2024-09-10T17:52:36.717 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:36.718 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:52:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:52:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:36.718 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:37.199 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:38.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:38.200 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:38.200 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:52:38.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:52:38.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:38.201 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:38.626 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:39.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:39.628 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:52:39.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:39.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:52:39.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:39.629 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:40.189 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:41.190 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:41.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:41.191 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:52:41.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:52:41.191 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:41.192 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:41.579 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:42.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:42.580 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:42.581 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:52:42.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:52:42.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:42.581 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:42.972 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:43.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:43.973 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:43.974 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:52:43.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:52:43.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:43.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:44.468 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:45.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:45.469 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:45.470 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:52:45.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:52:45.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:45.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:45.900 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:46.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:46.902 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:46.902 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:52:46.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:52:46.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:46.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:47.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:48.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:48.298 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:48.298 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:52:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:52:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:48.299 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:48.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:49.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:49.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:49.958 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:52:49.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:52:49.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:49.958 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:50.696 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:51.697 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:51.698 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:51.698 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:52:51.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:52:51.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:51.699 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:52.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:53.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:53.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:53.445 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:52:53.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:52:53.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:53.446 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:54.526 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:55.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:55.527 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:55.527 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:52:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:52:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:55.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:56.317 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:57.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:57.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:57.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:52:57.319 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:52:57.319 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:57.320 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:58.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:52:59.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:52:59.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:52:59.114 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:52:59.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:52:59.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:52:59.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T17:52:59.828 INFO:tasks.workunit.client.0.smithi086.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2939282618,v1:127.0.0.1:6803/2939282618] [v2:127.0.0.1:6804/2939282618,v1:127.0.0.1:6805/2939282618] exists,up f582df02-ae09-423f-bb1f-90d4cf62161c 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:52:59.832 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:52:59.833 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:52:59.834 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1abd65a9-173b-4da7-a067-913ae5858ba3 2024-09-10T17:52:59.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1abd65a9-173b-4da7-a067-913ae5858ba3' 2024-09-10T17:52:59.836 INFO:tasks.workunit.client.0.smithi086.stdout:add osd1 1abd65a9-173b-4da7-a067-913ae5858ba3 2024-09-10T17:52:59.836 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:52:59.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7h+Bm8YKhMhAA22BQnZgeUOZrdV3sBf0Rjg== 2024-09-10T17:52:59.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7h+Bm8YKhMhAA22BQnZgeUOZrdV3sBf0Rjg=="}' 2024-09-10T17:52:59.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1abd65a9-173b-4da7-a067-913ae5858ba3 -i td/test-erasure-eio/1/new.json 2024-09-10T17:53:00.353 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:53:00.364 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-10T17:53:00.365 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7h+Bm8YKhMhAA22BQnZgeUOZrdV3sBf0Rjg== --osd-uuid 1abd65a9-173b-4da7-a067-913ae5858ba3 2024-09-10T17:53:00.394 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:00.393+0000 7fbd89018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:00.404 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:00.403+0000 7fbd89018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:00.405 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:00.404+0000 7fbd89018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:00.406 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:00.405+0000 7fbd89018640 -1 bdev(0x5599ea64bc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2024-09-10T17:53:00.406 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:00.405+0000 7fbd89018640 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-10T17:53:07.098 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-10T17:53:07.099 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:53:07.099 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd1 key to auth repository 2024-09-10T17:53:07.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T17:53:07.100 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:53:07.626 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.1 2024-09-10T17:53:07.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T17:53:07.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:53:07.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:53:07.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:53:07.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:53:07.653 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:07.652+0000 7fb04cfb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:07.674 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:07.673+0000 7fb04cfb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:07.675 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:07.674+0000 7fb04cfb9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:08.049 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:08.050 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:53:08.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:53:08.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:08.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:08.458 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:09.264 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:09.263+0000 7fb04cfb9640 -1 Falling back to public interface 2024-09-10T17:53:09.459 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:09.460 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:09.460 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:53:09.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:53:09.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:09.461 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:09.884 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:10.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:10.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:10.886 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:53:10.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:53:10.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:10.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:10.961 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:10.960+0000 7fb04cfb9640 -1 osd.1 0 log_to_monitors true 2024-09-10T17:53:11.312 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:12.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:12.314 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:12.314 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:53:12.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:53:12.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:12.315 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:12.707 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:13.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:13.708 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:13.708 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:53:13.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:53:13.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:13.709 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:14.103 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:15.104 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:15.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:15.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:53:15.105 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:53:15.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:15.105 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:15.545 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:16.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:16.546 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:16.547 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:53:16.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:53:16.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:16.547 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:17.150 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:18.151 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:18.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:18.152 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:53:18.152 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:53:18.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:18.153 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:18.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:19.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:19.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:19.558 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:53:19.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:53:19.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:19.559 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:20.045 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:21.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:21.047 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:21.047 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:53:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:53:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:21.048 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:21.542 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:22.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:22.543 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:22.544 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:53:22.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:53:22.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:22.544 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:23.064 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:24.065 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:24.066 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:53:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:53:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:24.066 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:25.051 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:26.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:26.053 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:26.053 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:53:26.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:53:26.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:26.054 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:26.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:27.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:27.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:27.914 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:53:27.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:53:27.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:27.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:28.765 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:29.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:29.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:29.767 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:53:29.767 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:53:29.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:29.768 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:30.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:31.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:31.628 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:53:31.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:53:31.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:31.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:32.288 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:33.289 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:33.290 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:33.290 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:53:33.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:53:33.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:33.291 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:34.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:35.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:35.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:35.443 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:53:35.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:53:35.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:35.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T17:53:35.974 INFO:tasks.workunit.client.0.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1828094747,v1:127.0.0.1:6811/1828094747] [v2:127.0.0.1:6812/1828094747,v1:127.0.0.1:6813/1828094747] exists,up 1abd65a9-173b-4da7-a067-913ae5858ba3 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:53:35.979 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:53:35.980 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:53:35.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=16639cc6-b159-4998-af58-47801c1ee7e2 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 16639cc6-b159-4998-af58-47801c1ee7e2' 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stdout:add osd2 16639cc6-b159-4998-af58-47801c1ee7e2 2024-09-10T17:53:35.982 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:53:35.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCfh+Bmpx5fOxAA64giZTNJqSDR9V+gYD1Aaw== 2024-09-10T17:53:35.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCfh+Bmpx5fOxAA64giZTNJqSDR9V+gYD1Aaw=="}' 2024-09-10T17:53:35.997 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 16639cc6-b159-4998-af58-47801c1ee7e2 -i td/test-erasure-eio/2/new.json 2024-09-10T17:53:36.490 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:53:36.501 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-10T17:53:36.502 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfh+Bmpx5fOxAA64giZTNJqSDR9V+gYD1Aaw== --osd-uuid 16639cc6-b159-4998-af58-47801c1ee7e2 2024-09-10T17:53:36.532 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:36.530+0000 7fdabe6e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:36.590 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:36.589+0000 7fdabe6e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:36.591 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:36.590+0000 7fdabe6e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:36.592 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:36.591+0000 7fdabe6e8640 -1 bdev(0x55a9842f3c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2024-09-10T17:53:36.592 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:36.591+0000 7fdabe6e8640 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-10T17:53:42.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-10T17:53:42.497 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:53:42.498 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd2 key to auth repository 2024-09-10T17:53:42.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T17:53:42.498 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:53:43.027 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:53:43.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T17:53:43.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:53:43.027 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:53:43.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:53:43.028 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:53:43.054 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:43.053+0000 7f2c65818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:43.081 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:43.080+0000 7f2c65818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:43.082 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:43.081+0000 7f2c65818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:53:43.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T17:53:43.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:53:43.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:53:43.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:53:43.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:53:43.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:43.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:53:43.413 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:53:43.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:43.413 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:43.827 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:44.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:44.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:44.829 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:53:44.829 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:53:44.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:44.830 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:44.922 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:44.921+0000 7f2c65818640 -1 Falling back to public interface 2024-09-10T17:53:45.262 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:46.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:46.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:46.264 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:53:46.264 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:53:46.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:46.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:46.498 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:53:46.497+0000 7f2c65818640 -1 osd.2 0 log_to_monitors true 2024-09-10T17:53:46.732 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:47.733 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:47.733 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:53:47.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:47.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:53:47.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:47.734 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:48.128 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:49.129 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:49.129 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:53:49.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:49.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:53:49.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:49.130 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:49.633 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:50.634 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:50.635 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:53:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:53:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:51.107 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:52.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:52.108 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:52.109 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:53:52.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:53:52.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:52.109 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:52.562 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:53.563 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:53.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:53.564 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:53:53.564 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:53:53.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:53.565 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:53.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:54.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:54.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:54.980 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:53:54.981 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:53:54.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:54.981 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:55.474 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:56.476 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:56.476 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:53:56.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:53:56.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:56.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:56.928 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:57.929 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:57.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:57.930 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:53:57.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:53:57.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:57.930 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:53:58.390 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:53:59.391 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:53:59.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:53:59.392 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:53:59.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:53:59.392 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:53:59.393 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:00.605 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:01.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:01.606 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:01.607 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:54:01.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:54:01.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:01.607 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:02.523 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:03.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:03.524 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:03.524 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:54:03.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:54:03.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:03.525 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:04.400 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:05.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:05.401 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:05.402 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:54:05.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:54:05.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:05.402 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:06.622 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:07.623 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:07.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:07.624 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:54:07.624 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:54:07.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:07.625 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:08.112 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:09.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:09.113 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:09.113 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:54:09.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:54:09.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:09.114 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:09.504 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173437875,v1:127.0.0.1:6819/173437875] [v2:127.0.0.1:6820/173437875,v1:127.0.0.1:6821/173437875] exists,up 16639cc6-b159-4998-af58-47801c1ee7e2 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T17:54:09.508 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:54:09.509 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T17:54:09.511 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:54:09.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1dd133bc-9d84-48a0-aeb4-4e43515aa32a 2024-09-10T17:54:09.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1dd133bc-9d84-48a0-aeb4-4e43515aa32a' 2024-09-10T17:54:09.514 INFO:tasks.workunit.client.0.smithi086.stdout:add osd3 1dd133bc-9d84-48a0-aeb4-4e43515aa32a 2024-09-10T17:54:09.514 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T17:54:09.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBh+BmD/BRHxAAiwriK6SVFHf1TpH7es78Qw== 2024-09-10T17:54:09.528 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBh+BmD/BRHxAAiwriK6SVFHf1TpH7es78Qw=="}' 2024-09-10T17:54:09.529 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1dd133bc-9d84-48a0-aeb4-4e43515aa32a -i td/test-erasure-eio/3/new.json 2024-09-10T17:54:10.666 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:54:10.677 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-10T17:54:10.678 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBh+BmD/BRHxAAiwriK6SVFHf1TpH7es78Qw== --osd-uuid 1dd133bc-9d84-48a0-aeb4-4e43515aa32a 2024-09-10T17:54:10.708 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:10.706+0000 7fb71a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:10.709 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:10.708+0000 7fb71a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:10.710 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:10.709+0000 7fb71a81e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:10.711 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:10.710+0000 7fb71a81e640 -1 bdev(0x56016282bc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2024-09-10T17:54:10.711 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:10.710+0000 7fb71a81e640 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-10T17:54:17.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-10T17:54:17.115 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T17:54:17.116 INFO:tasks.workunit.client.0.smithi086.stdout:adding osd3 key to auth repository 2024-09-10T17:54:17.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T17:54:17.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T17:54:17.646 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.3 2024-09-10T17:54:17.646 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T17:54:17.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:54:17.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T17:54:17.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T17:54:17.647 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T17:54:17.673 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:17.672+0000 7f231c20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:17.696 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:17.695+0000 7f231c20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:17.697 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:17.696+0000 7f231c20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:54:18.074 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:54:18.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:18.075 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:18.494 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:19.048 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:19.047+0000 7f231c20d640 -1 Falling back to public interface 2024-09-10T17:54:19.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:19.495 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:19.496 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:54:19.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:54:19.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:19.496 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:19.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:20.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:20.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:20.922 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:54:20.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:54:20.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:20.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:20.982 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:20.981+0000 7f231c20d640 -1 osd.3 0 log_to_monitors true 2024-09-10T17:54:21.326 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:22.327 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:22.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:22.328 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:54:22.328 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:54:22.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:22.329 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:23.117 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:24.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:24.118 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:24.119 INFO:tasks.workunit.client.0.smithi086.stdout:4 2024-09-10T17:54:24.119 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T17:54:24.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:24.120 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:24.513 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:25.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:25.514 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:25.514 INFO:tasks.workunit.client.0.smithi086.stdout:5 2024-09-10T17:54:25.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T17:54:25.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:25.515 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:25.955 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:26.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:26.956 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:26.957 INFO:tasks.workunit.client.0.smithi086.stdout:6 2024-09-10T17:54:26.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T17:54:26.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:26.957 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:27.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:28.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:28.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:28.481 INFO:tasks.workunit.client.0.smithi086.stdout:7 2024-09-10T17:54:28.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T17:54:28.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:28.481 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:28.974 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:29.975 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:29.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:29.976 INFO:tasks.workunit.client.0.smithi086.stdout:8 2024-09-10T17:54:29.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T17:54:29.976 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:29.977 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:30.396 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:31.397 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:31.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:31.398 INFO:tasks.workunit.client.0.smithi086.stdout:9 2024-09-10T17:54:31.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T17:54:31.398 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:31.399 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:31.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:32.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:32.906 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:32.906 INFO:tasks.workunit.client.0.smithi086.stdout:10 2024-09-10T17:54:32.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T17:54:32.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:32.907 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:33.412 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:34.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:34.414 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:34.414 INFO:tasks.workunit.client.0.smithi086.stdout:11 2024-09-10T17:54:34.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T17:54:34.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:34.415 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:34.835 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:35.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:35.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:35.837 INFO:tasks.workunit.client.0.smithi086.stdout:12 2024-09-10T17:54:35.837 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T17:54:35.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:35.838 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:36.555 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:37.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:37.557 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:37.557 INFO:tasks.workunit.client.0.smithi086.stdout:13 2024-09-10T17:54:37.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T17:54:37.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:37.558 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:38.660 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:39.662 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:39.662 INFO:tasks.workunit.client.0.smithi086.stdout:14 2024-09-10T17:54:39.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T17:54:39.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:39.663 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:40.202 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:41.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:41.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:41.204 INFO:tasks.workunit.client.0.smithi086.stdout:15 2024-09-10T17:54:41.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T17:54:41.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:41.205 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:41.963 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:42.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:42.964 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:42.964 INFO:tasks.workunit.client.0.smithi086.stdout:16 2024-09-10T17:54:42.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T17:54:42.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:42.965 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:44.161 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:45.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:54:45.162 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:45.163 INFO:tasks.workunit.client.0.smithi086.stdout:17 2024-09-10T17:54:45.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T17:54:45.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:45.163 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T17:54:45.712 INFO:tasks.workunit.client.0.smithi086.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/324634142,v1:127.0.0.1:6827/324634142] [v2:127.0.0.1:6828/324634142,v1:127.0.0.1:6829/324634142] exists,up 1dd133bc-9d84-48a0-aeb4-4e43515aa32a 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:54:45.714 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:45.715 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:54:45.715 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.0.asok 2024-09-10T17:54:45.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-10T17:54:45.715 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.0.asok log flush 2024-09-10T17:54:45.793 INFO:tasks.workunit.client.0.smithi086.stdout:{} 2024-09-10T17:54:45.800 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-10T17:54:45.803 INFO:tasks.workunit.client.0.smithi086.stdout:2024-09-10T17:52:34.816+0000 7f1f2e824640 0 load: jerasure load: lrc 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:370: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-10T17:54:45.804 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-10T17:54:45.805 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-10T17:54:46.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-10T17:54:46.778 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-10T17:54:48.254 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' already exists 2024-09-10T17:54:48.266 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T17:54:49.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-10T17:54:49.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:54:49.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:54:49.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:54:49.269 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:54:49.371 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:54:49.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:54:49.758 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:54:49.758 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:54:49.758 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:54:49.758 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:54:49.759 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:54:49.759 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:54:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803805 2024-09-10T17:54:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803805 2024-09-10T17:54:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805' 2024-09-10T17:54:49.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:54:49.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:54:50.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574868 2024-09-10T17:54:50.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574868 2024-09-10T17:54:50.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574868' 2024-09-10T17:54:50.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:54:50.027 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:54:50.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=81604378643 2024-09-10T17:54:50.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 81604378643 2024-09-10T17:54:50.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574868 2-81604378643' 2024-09-10T17:54:50.138 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:54:50.138 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:54:50.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117005 2024-09-10T17:54:50.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117005 2024-09-10T17:54:50.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803805 1-55834574868 2-81604378643 3-115964117005' 2024-09-10T17:54:50.249 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:54:50.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803805 2024-09-10T17:54:50.250 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:54:50.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:54:50.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803805 2024-09-10T17:54:50.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:54:50.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803805 2024-09-10T17:54:50.252 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-10T17:54:50.252 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803805 2024-09-10T17:54:50.253 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:54:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803802 -lt 25769803805 2024-09-10T17:54:50.635 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:54:51.636 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:54:51.636 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:54:52.059 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-10T17:54:52.060 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:54:52.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574868 2024-09-10T17:54:52.060 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:54:52.061 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:54:52.061 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574868 2024-09-10T17:54:52.062 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:54:52.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574868 2024-09-10T17:54:52.063 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-09-10T17:54:52.063 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574868 2024-09-10T17:54:52.063 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:54:52.477 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574869 -lt 55834574868 2024-09-10T17:54:52.478 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:54:52.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-81604378643 2024-09-10T17:54:52.478 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:54:52.479 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:54:52.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-81604378643 2024-09-10T17:54:52.479 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:54:52.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=81604378643 2024-09-10T17:54:52.480 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 81604378643' 2024-09-10T17:54:52.481 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 81604378643 2024-09-10T17:54:52.481 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:54:52.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 81604378643 -lt 81604378643 2024-09-10T17:54:52.861 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:54:52.861 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117005 2024-09-10T17:54:52.861 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:54:52.862 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:54:52.863 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117005 2024-09-10T17:54:52.863 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:54:52.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117005 2024-09-10T17:54:52.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117005' 2024-09-10T17:54:52.864 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117005 2024-09-10T17:54:52.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:54:53.243 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117006 -lt 115964117005 2024-09-10T17:54:53.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:54:53.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:54:53.244 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:54:53.766 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:54:53.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:54:53.767 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:54:54.169 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:54:54.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:54:54.169 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:54:54.170 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-10T17:54:54.691 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-86744-1 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-86744-1 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-86744-1 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-10T17:54:54.692 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:54:54.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-10T17:54:54.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-10T17:54:54.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-10T17:54:54.693 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-86744-1 td/test-erasure-eio/ORIGINAL 2024-09-10T17:54:54.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-86744-1 td/test-erasure-eio 1 2024-09-10T17:54:54.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=ec 2024-09-10T17:54:54.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T17:54:54.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T17:54:54.849 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=pool-jerasure 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=obj-eio-86744-1 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/test-erasure-eio 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds pool-jerasure obj-eio-86744-1 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-1 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-1 2024-09-10T17:54:54.850 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:54:55.264 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('3' '1' '2') 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' ec '!=' ec ']' 2024-09-10T17:54:55.265 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/test-erasure-eio/1/type 2024-09-10T17:54:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T17:54:55.267 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:54:55.268 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:54:55.269 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.86744/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T17:54:55.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T17:54:55.352 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.86744/ceph-osd.1.asok 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T17:54:55.354 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.86744/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-86744-1 1 2024-09-10T17:54:55.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-86744-1 2024-09-10T17:54:55.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-10T17:54:55.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-10T17:54:55.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-86744-1 2024-09-10T17:54:55.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-10T17:54:55.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-10T17:54:55.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-86744-1 td/test-erasure-eio/COPY 2024-09-10T17:54:55.470 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-10T17:54:55.471 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-10T17:54:55.472 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-10T17:54:55.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-86744-1 2024-09-10T17:54:55.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:54:55.473 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-1 2024-09-10T17:54:55.473 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-1 2024-09-10T17:54:55.473 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:54:55.896 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:2' 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:54:55.897 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:54:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:54:56.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-10T17:54:57.013 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already out. 2024-09-10T17:54:57.025 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-86744-1 2024-09-10T17:54:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-10T17:54:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2024-09-10T17:54:57.026 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-86744-1 2024-09-10T17:54:57.026 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-86744-1 2024-09-10T17:54:57.026 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T17:54:57.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-10T17:54:57.713 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:54:57.713 INFO:tasks.workunit.client.0.smithi086.stderr:0' 2024-09-10T17:54:57.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2024-09-10T17:54:57.713 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-10T17:54:58.427 INFO:tasks.workunit.client.0.smithi086.stderr:osd.2 is already in. 2024-09-10T17:54:58.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-10T17:54:58.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T17:54:58.442 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:54:58.443 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' 2024-09-10T17:54:58.444 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T17:54:58.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-10T17:54:58.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-10T17:54:58.445 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T17:54:58.448 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-10T17:54:58.449 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T17:54:58.449 INFO:tasks.workunit.client.0.smithi086.stdout:start osd.2 2024-09-10T17:54:58.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=97891dfe-ca78-4304-94ae-ff380fc3b3e0 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.86744/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T17:54:58.450 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-10T17:54:58.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T17:54:58.450 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T17:54:58.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T17:54:58.451 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T17:54:58.468 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:58.467+0000 7f447aa1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:58.470 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:58.469+0000 7f447aa1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:58.471 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:58.470+0000 7f447aa1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T17:54:59.070 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T17:54:59.071 INFO:tasks.workunit.client.0.smithi086.stdout:0 2024-09-10T17:54:59.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:54:59.072 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:54:59.505 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:54:59.821 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:54:59.820+0000 7f447aa1d640 -1 Falling back to public interface 2024-09-10T17:55:00.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:00.506 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:00.506 INFO:tasks.workunit.client.0.smithi086.stdout:1 2024-09-10T17:55:00.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T17:55:00.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:00.507 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:55:00.903 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:01.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:01.904 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:01.905 INFO:tasks.workunit.client.0.smithi086.stdout:2 2024-09-10T17:55:01.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T17:55:01.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:01.905 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:55:02.294 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T17:55:02.353 INFO:tasks.workunit.client.0.smithi086.stderr:2024-09-10T17:55:02.352+0000 7f447aa1d640 -1 osd.2 35 log_to_monitors true 2024-09-10T17:55:03.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T17:55:03.295 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T17:55:03.296 INFO:tasks.workunit.client.0.smithi086.stdout:3 2024-09-10T17:55:03.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T17:55:03.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T17:55:03.296 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T17:55:03.917 INFO:tasks.workunit.client.0.smithi086.stdout:osd.2 up in weight 1 up_from 46 up_thru 19 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6818/725239875,v1:127.0.0.1:6819/725239875] [v2:127.0.0.1:6820/725239875,v1:127.0.0.1:6821/725239875] exists,up 16639cc6-b159-4998-af58-47801c1ee7e2 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T17:55:03.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T17:55:03.920 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T17:55:04.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T17:55:04.008 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T17:55:04.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T17:55:04.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T17:55:04.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T17:55:04.009 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T17:55:04.433 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:1 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:2 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:3' 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:55:04.512 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T17:55:04.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=25769803809 2024-09-10T17:55:04.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 25769803809 2024-09-10T17:55:04.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809' 2024-09-10T17:55:04.672 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:55:04.673 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T17:55:04.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574873 2024-09-10T17:55:04.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574873 2024-09-10T17:55:04.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574873' 2024-09-10T17:55:04.785 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:55:04.785 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T17:55:05.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-10T17:55:05.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-10T17:55:05.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574873 2-197568495619' 2024-09-10T17:55:05.091 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T17:55:05.091 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T17:55:05.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117010 2024-09-10T17:55:05.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117010 2024-09-10T17:55:05.203 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-25769803809 1-55834574873 2-197568495619 3-115964117010' 2024-09-10T17:55:05.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:55:05.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-25769803809 2024-09-10T17:55:05.204 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:55:05.204 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T17:55:05.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-25769803809 2024-09-10T17:55:05.205 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:55:05.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=25769803809 2024-09-10T17:55:05.206 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-10T17:55:05.206 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.0 seq 25769803809 2024-09-10T17:55:05.207 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T17:55:05.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-10T17:55:05.627 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:55:05.627 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-55834574873 2024-09-10T17:55:05.628 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:55:05.628 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T17:55:05.629 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-55834574873 2024-09-10T17:55:05.629 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:55:05.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574873 2024-09-10T17:55:05.630 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-10T17:55:05.630 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.1 seq 55834574873 2024-09-10T17:55:05.631 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:55:06.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574871 -lt 55834574873 2024-09-10T17:55:06.009 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T17:55:07.013 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T17:55:07.013 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T17:55:07.438 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574874 -lt 55834574873 2024-09-10T17:55:07.439 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:55:07.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-197568495619 2024-09-10T17:55:07.439 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:55:07.440 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T17:55:07.440 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-197568495619 2024-09-10T17:55:07.441 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:55:07.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-10T17:55:07.441 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-09-10T17:55:07.442 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.2 seq 197568495619 2024-09-10T17:55:07.442 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T17:55:07.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-10T17:55:07.864 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T17:55:07.864 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-115964117010 2024-09-10T17:55:07.865 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T17:55:07.865 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T17:55:07.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-115964117010 2024-09-10T17:55:07.866 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T17:55:07.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117010 2024-09-10T17:55:07.867 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 115964117010' 2024-09-10T17:55:07.867 INFO:tasks.workunit.client.0.smithi086.stdout:waiting osd.3 seq 115964117010 2024-09-10T17:55:07.868 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T17:55:08.250 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117011 -lt 115964117010 2024-09-10T17:55:08.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T17:55:08.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:55:08.251 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:55:08.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-10T17:55:08.770 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T17:55:08.771 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T17:55:09.160 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-10T17:55:09.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T17:55:09.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T17:55:09.161 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T17:55:09.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-10T17:55:09.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T17:55:09.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T17:55:09.680 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-10T17:55:09.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-10T17:55:09.681 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-10T17:55:10.265 INFO:tasks.workunit.client.0.smithi086.stderr:pool 'pool-jerasure' does not exist 2024-09-10T17:55:10.276 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-10T17:55:10.765 INFO:tasks.workunit.client.0.smithi086.stderr:erasure-code-profile myprofile does not exist 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:55:10.777 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:55:10.885 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:55:10.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:55:10.886 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:55:10.886 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:55:10.887 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:55:10.888 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:55:10.888 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:55:10.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:10.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:55:10.889 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:55:10.889 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:10.890 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:10.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T17:55:10.891 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:55:10.911 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:55:10.911 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:10.912 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:55:10.912 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:55:10.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:55:10.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:55:10.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T17:55:10.913 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/test-erasure-eio 0 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T17:55:10.914 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T17:55:10.915 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T17:55:10.915 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T17:55:10.916 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T17:55:10.916 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T17:55:10.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T17:55:10.918 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T17:55:10.918 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T17:55:10.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:10.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T17:55:10.919 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T17:55:10.919 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T17:55:10.920 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:10.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T17:55:10.921 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2024-09-10T17:55:10.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T17:55:10.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T17:55:10.922 INFO:tasks.workunit.client.0.smithi086.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.86744 2024-09-10T17:55:10.922 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.86744 2024-09-10T17:55:10.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T17:55:10.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T17:55:10.923 INFO:tasks.workunit.client.0.smithi086.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T17:55:10.925 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T17:55:10.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T17:55:10.950 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-10T17:55:10.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T17:55:11.270 DEBUG:teuthology.parallel:result is None 2024-09-10T17:55:11.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T17:55:11.298 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T17:55:11.298 DEBUG:teuthology.orchestra.run.smithi086:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T17:55:11.363 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T17:55:11.363 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T17:55:11.376 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T17:55:11.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T17:55:11.452 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T17:55:11.461 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T17:55:11.461 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T17:55:11.482 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-09-10T17:55:11.487 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T17:55:11.487 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-09-10T17:55:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv01.front.sepia.ceph.com 3 7 377 8 +255us[ +268us] +/- 55ms 2024-09-10T17:55:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 130 +383us[ +393us] +/- 65ms 2024-09-10T17:55:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -867us[ -867us] +/- 67ms 2024-09-10T17:55:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T17:55:11.489 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T17:55:11.497 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T17:55:11.497 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T17:55:11.507 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T17:55:11.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T17:55:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:55:11.615 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T17:55:11.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T17:55:11.633 INFO:teuthology.task.internal:Duration was 4888.987152 seconds 2024-09-10T17:55:11.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T17:55:11.645 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T17:55:11.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T17:55:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T17:55:11.871 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T17:55:11.871 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-09-10T17:55:11.872 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T17:55:11.936 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T17:55:11.937 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T17:55:12.003 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T17:55:12.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T17:55:12.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T17:55:12.151 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T17:55:12.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T17:55:12.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T17:55:12.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:12.239 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-09-10T17:55:12.252 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T17:55:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T17:55:12.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T17:55:12.320 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T17:55:12.322 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899327/remote/smithi086 2024-09-10T17:55:12.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T17:55:12.394 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T17:55:12.394 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T17:55:12.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T17:55:12.453 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T17:55:12.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T17:55:12.461 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T17:55:12.461 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T17:55:12.498 INFO:teuthology.orchestra.run.smithi086.stdout: 266054 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 17:55 /home/ubuntu/cephtest 2024-09-10T17:55:12.499 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T17:55:12.509 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T17:55:12.534 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} duration: 4888.987151622772 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-10T17:55:12.534 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T17:55:12.635 INFO:teuthology.run:pass